developer_uid: rirv938
submission_id: rirv938-20250411-reward_33516_v1
model_name: rirv938-20250411-reward_33516_v1
model_group: rirv938/20250411_reward_
status: torndown
timestamp: 2025-04-12T07:38:19+00:00
num_battles: 14202
num_wins: 7539
celo_rating: 1311.1
family_friendly_score: 0.5564
family_friendly_standard_error: 0.007025938229161996
submission_type: basic
model_repo: rirv938/20250411_reward_safety_1_2_removed
model_architecture: LlamaForSequenceClassification
model_num_parameters: 8030261248.0
best_of: 1
max_input_tokens: 512
max_output_tokens: 1
reward_model: default
display_name: rirv938-20250411-reward_33516_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: rirv938/20250411_reward_safety_1_2_removed
model_size: 8B
ranking_group: single
us_pacific_date: 2025-04-12
win_ratio: 0.5308407266582171
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-20250411-reward-33516-v1-mkmlizer
Waiting for job on rirv938-20250411-reward-33516-v1-mkmlizer to finish
rirv938-20250411-reward-33516-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-20250411-reward-33516-v1-mkmlizer: ║     _____            __           __                                ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║       /___/                                                         ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║                                                                     ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  Version: 0.12.8                                                    ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║                                                                     ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║                                                                     ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
rirv938-20250411-reward-33516-v1-mkmlizer: ║                                                                     ║
rirv938-20250411-reward-33516-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-20250411-reward-33516-v1-mkmlizer: Downloaded to shared memory in 27.946s
rirv938-20250411-reward-33516-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmp1emcefoo, device:0
rirv938-20250411-reward-33516-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission chaiml-mistral31-ff4kex_37571_v2: HTTPConnectionPool(host='chaiml-mistral31-ff4kex-37571-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
rirv938-20250411-reward-33516-v1-mkmlizer: quantized model in 84.574s
rirv938-20250411-reward-33516-v1-mkmlizer: Processed model rirv938/20250411_reward_safety_1_2_removed in 112.520s
rirv938-20250411-reward-33516-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-20250411-reward-33516-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-20250411-reward-33516-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v1
rirv938-20250411-reward-33516-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v1/config.json
rirv938-20250411-reward-33516-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v1/special_tokens_map.json
rirv938-20250411-reward-33516-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v1/tokenizer_config.json
rirv938-20250411-reward-33516-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v1/flywheel_model.0.safetensors
rirv938-20250411-reward-33516-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:55,  5.21it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:30,  3.17it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:02,  2.34it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:03,  4.49it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:01,  4.56it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:54,  5.19it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:04,  4.33it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:25,  3.25it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:49,  2.54it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:02,  4.36it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:59,  4.61it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:52,  5.15it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:01,  4.37it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:21,  3.31it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:43,  2.58it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:00,  4.40it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:56,  4.64it/s]
Loading 0:  10%|▉         | 28/291 [00:06<00:50,  5.23it/s]
Loading 0:  10%|█         | 30/291 [00:07<00:59,  4.40it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:18,  3.32it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:39,  2.60it/s]
Loading 0:  12%|█▏        | 35/291 [00:08<00:57,  4.41it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:54,  4.65it/s]
Loading 0:  13%|█▎        | 37/291 [00:09<00:48,  5.23it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:57,  4.38it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:15,  3.30it/s]
Loading 0:  14%|█▍        | 41/291 [00:11<01:36,  2.59it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:56,  4.40it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:52,  4.64it/s]
Loading 0:  16%|█▌        | 46/291 [00:11<00:46,  5.23it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<00:55,  4.41it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:12,  3.34it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:31,  2.63it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:53,  4.45it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:50,  4.69it/s]
Loading 0:  19%|█▉        | 55/291 [00:13<00:44,  5.26it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:52,  4.44it/s]
Loading 0:  20%|█▉        | 58/291 [00:14<01:09,  3.35it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:27,  2.64it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:51,  4.47it/s]
Loading 0:  22%|██▏       | 63/291 [00:15<00:48,  4.71it/s]
Loading 0:  22%|██▏       | 64/291 [00:16<00:43,  5.18it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:51,  4.40it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:07,  3.33it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:25,  2.62it/s]
Loading 0:  24%|██▍       | 71/291 [00:18<00:49,  4.42it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:46,  4.66it/s]
Loading 0:  25%|██▌       | 73/291 [00:18<00:42,  5.17it/s]
Loading 0:  26%|██▌       | 75/291 [00:18<00:49,  4.39it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<01:04,  3.33it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<01:21,  2.62it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:47,  4.44it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:44,  4.67it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:40,  5.18it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:39,  5.33it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:58,  3.54it/s]
Loading 0:  29%|██▉       | 85/291 [00:21<01:13,  2.80it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:29,  2.29it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:48,  4.20it/s]
Loading 0:  31%|███       | 90/291 [00:22<00:44,  4.49it/s]
Loading 0:  31%|███▏      | 91/291 [00:23<00:39,  5.08it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:45,  4.34it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<01:00,  3.28it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:15,  2.59it/s]
Loading 0:  34%|███▎      | 98/291 [00:24<00:43,  4.43it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:41,  4.67it/s]
Loading 0:  34%|███▍      | 100/291 [00:25<00:36,  5.25it/s]
Loading 0:  35%|███▌      | 102/291 [00:25<00:42,  4.44it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:56,  3.35it/s]
Loading 0:  36%|███▌      | 104/291 [00:27<01:11,  2.63it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:40,  4.51it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:38,  4.75it/s]
Loading 0:  37%|███▋      | 109/291 [00:27<00:34,  5.32it/s]
Loading 0:  38%|███▊      | 111/291 [00:28<00:40,  4.48it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:53,  3.38it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:07,  2.64it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:39,  4.48it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:36,  4.71it/s]
Loading 0:  41%|████      | 118/291 [00:29<00:32,  5.29it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:38,  4.45it/s]
Loading 0:  42%|████▏     | 121/291 [00:30<00:50,  3.37it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<01:03,  2.65it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:36,  4.49it/s]
Loading 0:  43%|████▎     | 126/291 [00:31<00:34,  4.72it/s]
Loading 0:  44%|████▎     | 127/291 [00:31<00:30,  5.30it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:36,  4.46it/s]
Loading 0:  45%|████▍     | 130/291 [00:33<00:47,  3.37it/s]
Loading 0:  45%|████▌     | 131/291 [00:33<01:00,  2.63it/s]
Loading 0:  46%|████▌     | 134/291 [00:33<00:35,  4.47it/s]
Loading 0:  46%|████▋     | 135/291 [00:34<00:33,  4.71it/s]
Loading 0:  47%|████▋     | 136/291 [00:34<00:29,  5.29it/s]
Loading 0:  47%|████▋     | 138/291 [00:34<00:34,  4.47it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:44,  3.38it/s]
Loading 0:  48%|████▊     | 140/291 [00:36<00:56,  2.66it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:32,  4.49it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:31,  4.73it/s]
Loading 0:  50%|████▉     | 145/291 [00:36<00:27,  5.24it/s]
Loading 0:  51%|█████     | 147/291 [00:37<00:32,  4.45it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:42,  3.37it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:53,  2.65it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:31,  4.48it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:29,  4.73it/s]
Loading 0:  53%|█████▎    | 154/291 [00:38<00:25,  5.29it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:30,  4.47it/s]
Loading 0:  54%|█████▍    | 157/291 [00:39<00:39,  3.37it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:49,  2.67it/s]
Loading 0:  55%|█████▌    | 161/291 [00:40<00:28,  4.57it/s]
Loading 0:  56%|█████▌    | 162/291 [00:40<00:26,  4.81it/s]
Loading 0:  56%|█████▌    | 163/291 [00:40<00:23,  5.40it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:27,  4.51it/s]
Loading 0:  57%|█████▋    | 166/291 [00:42<00:36,  3.40it/s]
Loading 0:  57%|█████▋    | 167/291 [00:42<00:46,  2.69it/s]
Loading 0:  58%|█████▊    | 170/291 [00:42<00:26,  4.62it/s]
Loading 0:  59%|█████▉    | 171/291 [00:43<00:24,  4.85it/s]
Loading 0:  59%|█████▉    | 172/291 [00:43<00:21,  5.43it/s]
Loading 0:  59%|█████▉    | 173/291 [00:43<00:32,  3.65it/s]
Loading 0:  60%|██████    | 175/291 [00:43<00:23,  4.96it/s]
Loading 0:  60%|██████    | 176/291 [00:44<00:22,  5.18it/s]
Loading 0:  61%|██████    | 177/291 [00:44<00:19,  5.74it/s]
Loading 0:  62%|██████▏   | 179/291 [00:44<00:24,  4.58it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:32,  3.38it/s]
Loading 0:  62%|██████▏   | 181/291 [00:45<00:41,  2.66it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:23,  4.64it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:21,  4.87it/s]
Loading 0:  64%|██████▍   | 186/291 [00:46<00:19,  5.46it/s]
Loading 0:  64%|██████▍   | 187/291 [00:46<00:18,  5.58it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:28,  3.62it/s]
Loading 0:  65%|██████▍   | 189/291 [00:47<00:38,  2.67it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:27,  3.64it/s]
Loading 0:  66%|██████▋   | 193/291 [00:48<00:32,  3.00it/s]
Loading 0:  67%|██████▋   | 194/291 [00:49<00:39,  2.48it/s]
Loading 0:  68%|██████▊   | 197/291 [00:49<00:22,  4.20it/s]
Loading 0:  68%|██████▊   | 198/291 [00:49<00:20,  4.48it/s]
Loading 0:  68%|██████▊   | 199/291 [00:49<00:18,  5.02it/s]
Loading 0:  69%|██████▉   | 201/291 [00:50<00:20,  4.35it/s]
Loading 0:  69%|██████▉   | 202/291 [00:51<00:26,  3.32it/s]
Loading 0:  70%|██████▉   | 203/291 [00:51<00:33,  2.65it/s]
Loading 0:  71%|███████   | 206/291 [00:51<00:18,  4.49it/s]
Loading 0:  71%|███████   | 207/291 [00:52<00:17,  4.74it/s]
Loading 0:  71%|███████▏  | 208/291 [00:52<00:15,  5.31it/s]
Loading 0:  72%|███████▏  | 210/291 [00:52<00:18,  4.49it/s]
Loading 0:  73%|███████▎  | 211/291 [00:53<00:23,  3.39it/s]
Loading 0:  73%|███████▎  | 212/291 [00:53<00:29,  2.67it/s]
Loading 0:  74%|███████▍  | 215/291 [00:54<00:16,  4.52it/s]
Loading 0:  74%|███████▍  | 216/291 [00:54<00:15,  4.76it/s]
Loading 0:  75%|███████▍  | 217/291 [00:54<00:13,  5.31it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:16,  4.48it/s]
Loading 0:  76%|███████▌  | 220/291 [00:55<00:20,  3.38it/s]
Loading 0:  76%|███████▌  | 221/291 [00:56<00:26,  2.68it/s]
Loading 0:  77%|███████▋  | 224/291 [00:56<00:14,  4.58it/s]
Loading 0:  77%|███████▋  | 225/291 [00:56<00:13,  4.83it/s]
Loading 0:  78%|███████▊  | 226/291 [00:56<00:12,  5.37it/s]
Loading 0:  78%|███████▊  | 228/291 [00:57<00:14,  4.50it/s]
Loading 0:  79%|███████▊  | 229/291 [00:57<00:18,  3.39it/s]
Loading 0:  79%|███████▉  | 230/291 [00:58<00:22,  2.68it/s]
Loading 0:  80%|████████  | 233/291 [00:58<00:12,  4.59it/s]
Loading 0:  80%|████████  | 234/291 [00:58<00:11,  4.82it/s]
Loading 0:  81%|████████  | 235/291 [00:58<00:10,  5.34it/s]
Loading 0:  81%|████████▏ | 237/291 [00:59<00:12,  4.48it/s]
Loading 0:  82%|████████▏ | 238/291 [01:00<00:15,  3.38it/s]
Loading 0:  82%|████████▏ | 239/291 [01:00<00:19,  2.67it/s]
Loading 0:  83%|████████▎ | 242/291 [01:00<00:10,  4.53it/s]
Loading 0:  84%|████████▎ | 243/291 [01:01<00:10,  4.77it/s]
Loading 0:  84%|████████▍ | 244/291 [01:01<00:08,  5.33it/s]
Loading 0:  85%|████████▍ | 246/291 [01:01<00:10,  4.48it/s]
Loading 0:  85%|████████▍ | 247/291 [01:02<00:13,  3.37it/s]
Loading 0:  85%|████████▌ | 248/291 [01:02<00:16,  2.68it/s]
Loading 0:  86%|████████▋ | 251/291 [01:03<00:08,  4.54it/s]
Loading 0:  87%|████████▋ | 252/291 [01:03<00:08,  4.79it/s]
Loading 0:  87%|████████▋ | 253/291 [01:03<00:07,  5.34it/s]
Loading 0:  88%|████████▊ | 255/291 [01:03<00:08,  4.48it/s]
Loading 0:  88%|████████▊ | 256/291 [01:04<00:10,  3.39it/s]
Loading 0:  88%|████████▊ | 257/291 [01:05<00:12,  2.68it/s]
Loading 0:  89%|████████▉ | 260/291 [01:05<00:06,  4.60it/s]
Loading 0:  90%|████████▉ | 261/291 [01:05<00:06,  4.83it/s]
Loading 0:  90%|█████████ | 262/291 [01:05<00:05,  5.38it/s]
Loading 0:  91%|█████████ | 264/291 [01:06<00:05,  4.51it/s]
Loading 0:  91%|█████████ | 265/291 [01:06<00:07,  3.39it/s]
Loading 0:  91%|█████████▏| 266/291 [01:07<00:09,  2.64it/s]
Loading 0:  92%|█████████▏| 269/291 [01:07<00:04,  4.54it/s]
Loading 0:  93%|█████████▎| 270/291 [01:07<00:04,  4.79it/s]
Loading 0:  93%|█████████▎| 271/291 [01:07<00:03,  5.37it/s]
Loading 0:  94%|█████████▍| 273/291 [01:08<00:03,  4.51it/s]
Loading 0:  94%|█████████▍| 274/291 [01:08<00:04,  3.40it/s]
Loading 0:  95%|█████████▍| 275/291 [01:09<00:05,  2.68it/s]
Loading 0:  96%|█████████▌| 278/291 [01:09<00:02,  4.59it/s]
Loading 0:  96%|█████████▌| 279/291 [01:09<00:02,  4.82it/s]
Loading 0:  96%|█████████▌| 280/291 [01:09<00:02,  5.39it/s]
Loading 0:  97%|█████████▋| 281/291 [01:10<00:02,  3.64it/s]
Loading 0:  97%|█████████▋| 282/291 [01:11<00:03,  2.75it/s]
Loading 0:  98%|█████████▊| 284/291 [01:11<00:01,  3.94it/s]
Loading 0:  98%|█████████▊| 285/291 [01:11<00:01,  4.30it/s]
Loading 0:  98%|█████████▊| 286/291 [01:11<00:01,  4.99it/s]
Loading 0:  99%|█████████▊| 287/291 [01:11<00:00,  5.27it/s]
Loading 0:  99%|█████████▉| 288/291 [01:12<00:00,  3.43it/s]
                                                            

Job rirv938-20250411-reward-33516-v1-mkmlizer completed after 136.28s with status: succeeded
Stopping job with name rirv938-20250411-reward-33516-v1-mkmlizer
Pipeline stage MKMLizer completed in 136.75s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-20250411-reward-33516-v1
Waiting for inference service rirv938-20250411-reward-33516-v1 to be ready
Failed to get response for submission chaiml-mistral31-ff4kex_37571_v2: HTTPConnectionPool(host='chaiml-mistral31-ff4kex-37571-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-mistral31-ff4kex_37571_v2: HTTPConnectionPool(host='chaiml-mistral31-ff4kex-37571-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service rirv938-20250411-reward-33516-v1 ready after 110.38112211227417s
Pipeline stage MKMLDeployer completed in 110.93s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.325005769729614s
Received healthy response to inference request in 7.272905588150024s
Received healthy response to inference request in 2.9011096954345703s
Received healthy response to inference request in 4.28017520904541s
Received healthy response to inference request in 2.1647603511810303s
5 requests
0 failed requests
5th percentile: 2.312030220031738
10th percentile: 2.459300088882446
20th percentile: 2.7538398265838624
30th percentile: 3.1769227981567383
40th percentile: 3.728549003601074
50th percentile: 4.28017520904541
60th percentile: 4.298107433319092
70th percentile: 4.316039657592773
80th percentile: 4.914585733413697
90th percentile: 6.0937456607818605
95th percentile: 6.683325624465942
99th percentile: 7.154989595413208
mean time: 4.18879132270813
%s, retrying in %s seconds...
Received healthy response to inference request in 2.265310525894165s
Received healthy response to inference request in 3.3855230808258057s
Received healthy response to inference request in 6.46632981300354s
Received healthy response to inference request in 5.415906190872192s
Received healthy response to inference request in 2.226309061050415s
5 requests
0 failed requests
5th percentile: 2.234109354019165
10th percentile: 2.241909646987915
20th percentile: 2.257510232925415
30th percentile: 2.4893530368804933
40th percentile: 2.9374380588531497
50th percentile: 3.3855230808258057
60th percentile: 4.19767632484436
70th percentile: 5.009829568862915
80th percentile: 5.625990915298462
90th percentile: 6.046160364151001
95th percentile: 6.256245088577271
99th percentile: 6.424312868118286
mean time: 3.9518757343292235
%s, retrying in %s seconds...
Received healthy response to inference request in 4.406403303146362s
Received healthy response to inference request in 3.254211902618408s
Received healthy response to inference request in 2.502415180206299s
Received healthy response to inference request in 3.3884716033935547s
Received healthy response to inference request in 2.1939032077789307s
5 requests
0 failed requests
5th percentile: 2.255605602264404
10th percentile: 2.3173079967498778
20th percentile: 2.4407127857208253
30th percentile: 2.652774524688721
40th percentile: 2.9534932136535645
50th percentile: 3.254211902618408
60th percentile: 3.307915782928467
70th percentile: 3.3616196632385256
80th percentile: 3.592057943344116
90th percentile: 3.9992306232452393
95th percentile: 4.202816963195801
99th percentile: 4.36568603515625
mean time: 3.1490810394287108
Pipeline stage StressChecker completed in 60.61s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.74s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.78s
Shutdown handler de-registered
rirv938-20250411-reward_33516_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250411-reward-33516-v1-profiler
Waiting for inference service rirv938-20250411-reward-33516-v1-profiler to be ready
Inference service rirv938-20250411-reward-33516-v1-profiler ready after 120.48486542701721s
Pipeline stage MKMLProfilerDeployer completed in 120.91s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250411-rewe4fa11bdf7cc91f47da131296a0cc811-deplocmmqw:/code/chaiverse_profiler_1744443980 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250411-rewe4fa11bdf7cc91f47da131296a0cc811-deplocmmqw --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1744443980 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1744443980/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v1-profiler is running
Tearing down inference service rirv938-20250411-reward-33516-v1-profiler
Service rirv938-20250411-reward-33516-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.45s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.38s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250411-reward-33516-v1-profiler
Waiting for inference service rirv938-20250411-reward-33516-v1-profiler to be ready
Inference service rirv938-20250411-reward-33516-v1-profiler ready after 120.46938824653625s
Pipeline stage MKMLProfilerDeployer completed in 120.82s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250411-rewe4fa11bdf7cc91f47da131296a0cc811-deplomj2gv:/code/chaiverse_profiler_1744447606 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250411-rewe4fa11bdf7cc91f47da131296a0cc811-deplomj2gv --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1744447606 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1744447606/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v1-profiler is running
Tearing down inference service rirv938-20250411-reward-33516-v1-profiler
Service rirv938-20250411-reward-33516-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.47s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.41s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250411-reward-33516-v1-profiler
Waiting for inference service rirv938-20250411-reward-33516-v1-profiler to be ready
Inference service rirv938-20250411-reward-33516-v1-profiler ready after 50.22878837585449s
Pipeline stage MKMLProfilerDeployer completed in 50.59s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250411-rewe4fa11bdf7cc91f47da131296a0cc811-deplo4lw56:/code/chaiverse_profiler_1744451171 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250411-rewe4fa11bdf7cc91f47da131296a0cc811-deplo4lw56 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1744451171 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1744451171/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v1-profiler is running
Tearing down inference service rirv938-20250411-reward-33516-v1-profiler
Service rirv938-20250411-reward-33516-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.43s
Shutdown handler de-registered
rirv938-20250411-reward_33516_v1 status is now inactive due to auto deactivation removed underperforming models
rirv938-20250411-reward_33516_v1 status is now torndown due to DeploymentManager action