rirv938-20250402-reward_22740

developer_uid: rirv938
submission_id: rirv938-20250402-reward_22740_v2
model_name: rirv938-20250402-reward_22740_v2
model_group: rirv938/20250402_reward_
status: torndown
timestamp: 2025-04-03T02:11:56+00:00
num_battles: 12330
num_wins: 6668
celo_rating: 1312.87
family_friendly_score: 0.5469999999999999
family_friendly_standard_error: 0.007039758518585705
submission_type: basic
model_repo: rirv938/20250402_reward_ava_cosine_2
model_architecture: LlamaForSequenceClassification
model_num_parameters: 8030261248.0
best_of: 1
max_input_tokens: 512
max_output_tokens: 1
reward_model: default
display_name: rirv938-20250402-reward_22740_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: rirv938/20250402_reward_ava_cosine_2
model_size: 8B
ranking_group: single
us_pacific_date: 2025-04-02
win_ratio: 0.5407948094079481
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-20250402-reward-22740-v2-mkmlizer
Waiting for job on rirv938-20250402-reward-22740-v2-mkmlizer to finish
rirv938-20250402-reward-22740-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-20250402-reward-22740-v2-mkmlizer: ║     _____            __           __                                ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║       /___/                                                         ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║                                                                     ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  Version: 0.12.8                                                    ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║                                                                     ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  belonging to:                                                      ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║                                                                     ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
rirv938-20250402-reward-22740-v2-mkmlizer: ║                                                                     ║
rirv938-20250402-reward-22740-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-20250402-reward-22740-v2-mkmlizer: Downloaded to shared memory in 24.651s
rirv938-20250402-reward-22740-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpfnn_82_z, device:0
rirv938-20250402-reward-22740-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission chaiml-sft-gemma2-28b-v_83370_v4: HTTPConnectionPool(host='chaiml-sft-gemma2-28b-v-83370-v4-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
rirv938-20250402-reward-22740-v2-mkmlizer: quantized model in 85.513s
rirv938-20250402-reward-22740-v2-mkmlizer: Processed model rirv938/20250402_reward_ava_cosine_2 in 110.165s
rirv938-20250402-reward-22740-v2-mkmlizer: creating bucket guanaco-mkml-models
rirv938-20250402-reward-22740-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-20250402-reward-22740-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-20250402-reward-22740-v2
rirv938-20250402-reward-22740-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-20250402-reward-22740-v2/config.json
rirv938-20250402-reward-22740-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-20250402-reward-22740-v2/special_tokens_map.json
rirv938-20250402-reward-22740-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-20250402-reward-22740-v2/tokenizer_config.json
rirv938-20250402-reward-22740-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-20250402-reward-22740-v2/tokenizer.json
rirv938-20250402-reward-22740-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-20250402-reward-22740-v2/flywheel_model.0.safetensors
rirv938-20250402-reward-22740-v2-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:54,  5.24it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:30,  3.18it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:01,  2.36it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:02,  4.52it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:01,  4.56it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:54,  5.12it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:04,  4.35it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:26,  3.23it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:49,  2.52it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:02,  4.35it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:59,  4.60it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:52,  5.18it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:01,  4.38it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:20,  3.32it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:43,  2.58it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:00,  4.40it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:56,  4.66it/s]
Loading 0:  10%|▉         | 28/291 [00:06<00:50,  5.24it/s]
Loading 0:  10%|█         | 30/291 [00:07<00:58,  4.45it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:17,  3.38it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:37,  2.65it/s]
Loading 0:  12%|█▏        | 35/291 [00:08<00:56,  4.49it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:53,  4.74it/s]
Loading 0:  13%|█▎        | 37/291 [00:09<00:48,  5.26it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:56,  4.45it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:14,  3.36it/s]
Loading 0:  14%|█▍        | 41/291 [00:10<01:35,  2.62it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:55,  4.47it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:52,  4.72it/s]
Loading 0:  16%|█▌        | 46/291 [00:11<00:46,  5.24it/s]
Loading 0:  16%|█▋        | 48/291 [00:11<00:54,  4.44it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:11,  3.36it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:31,  2.64it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:53,  4.49it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:49,  4.74it/s]
Loading 0:  19%|█▉        | 55/291 [00:13<00:45,  5.20it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:53,  4.41it/s]
Loading 0:  20%|█▉        | 58/291 [00:14<01:09,  3.35it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:28,  2.63it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:51,  4.48it/s]
Loading 0:  22%|██▏       | 63/291 [00:15<00:48,  4.70it/s]
Loading 0:  22%|██▏       | 64/291 [00:15<00:43,  5.25it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:50,  4.42it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:07,  3.31it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:26,  2.59it/s]
Loading 0:  24%|██▍       | 71/291 [00:17<00:50,  4.37it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:47,  4.60it/s]
Loading 0:  25%|██▌       | 73/291 [00:18<00:42,  5.19it/s]
Loading 0:  26%|██▌       | 75/291 [00:18<00:49,  4.37it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<01:05,  3.30it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<01:22,  2.58it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:47,  4.44it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:44,  4.68it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:40,  5.16it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:40,  5.16it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:59,  3.48it/s]
Loading 0:  29%|██▉       | 85/291 [00:21<01:15,  2.74it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:29,  2.28it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:47,  4.23it/s]
Loading 0:  31%|███       | 90/291 [00:22<00:44,  4.52it/s]
Loading 0:  31%|███▏      | 91/291 [00:22<00:39,  5.10it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:46,  4.24it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<01:01,  3.19it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:17,  2.52it/s]
Loading 0:  34%|███▎      | 98/291 [00:25<00:44,  4.38it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:41,  4.62it/s]
Loading 0:  34%|███▍      | 100/291 [00:25<00:36,  5.18it/s]
Loading 0:  35%|███▌      | 102/291 [00:25<00:43,  4.30it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:57,  3.25it/s]
Loading 0:  36%|███▌      | 104/291 [00:27<01:13,  2.55it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:42,  4.36it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:39,  4.61it/s]
Loading 0:  37%|███▋      | 109/291 [00:27<00:35,  5.16it/s]
Loading 0:  38%|███▊      | 111/291 [00:28<00:40,  4.40it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:53,  3.33it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:08,  2.59it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:39,  4.41it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:37,  4.66it/s]
Loading 0:  41%|████      | 118/291 [00:29<00:33,  5.18it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:39,  4.37it/s]
Loading 0:  42%|████▏     | 121/291 [00:31<00:51,  3.29it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<01:05,  2.59it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:37,  4.40it/s]
Loading 0:  43%|████▎     | 126/291 [00:32<00:35,  4.63it/s]
Loading 0:  44%|████▎     | 127/291 [00:32<00:31,  5.22it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:37,  4.33it/s]
Loading 0:  45%|████▍     | 130/291 [00:33<00:49,  3.26it/s]
Loading 0:  45%|████▌     | 131/291 [00:34<01:02,  2.56it/s]
Loading 0:  46%|████▌     | 134/291 [00:34<00:35,  4.39it/s]
Loading 0:  46%|████▋     | 135/291 [00:34<00:33,  4.62it/s]
Loading 0:  47%|████▋     | 136/291 [00:34<00:29,  5.20it/s]
Loading 0:  47%|████▋     | 138/291 [00:35<00:35,  4.35it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:46,  3.29it/s]
Loading 0:  48%|████▊     | 140/291 [00:36<00:58,  2.60it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:33,  4.48it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:31,  4.73it/s]
Loading 0:  50%|████▉     | 145/291 [00:36<00:27,  5.28it/s]
Loading 0:  51%|█████     | 147/291 [00:37<00:32,  4.43it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:42,  3.34it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:53,  2.65it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:30,  4.51it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:29,  4.74it/s]
Loading 0:  53%|█████▎    | 154/291 [00:39<00:26,  5.21it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:30,  4.41it/s]
Loading 0:  54%|█████▍    | 157/291 [00:40<00:40,  3.34it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:50,  2.64it/s]
Loading 0:  55%|█████▌    | 161/291 [00:40<00:29,  4.46it/s]
Loading 0:  56%|█████▌    | 162/291 [00:41<00:27,  4.68it/s]
Loading 0:  56%|█████▌    | 163/291 [00:41<00:24,  5.26it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:29,  4.32it/s]
Loading 0:  57%|█████▋    | 166/291 [00:42<00:38,  3.24it/s]
Loading 0:  57%|█████▋    | 167/291 [00:43<00:48,  2.55it/s]
Loading 0:  58%|█████▊    | 170/291 [00:43<00:27,  4.38it/s]
Loading 0:  59%|█████▉    | 171/291 [00:43<00:25,  4.63it/s]
Loading 0:  59%|█████▉    | 172/291 [00:43<00:22,  5.19it/s]
Loading 0:  59%|█████▉    | 173/291 [00:44<00:32,  3.58it/s]
Loading 0:  60%|██████    | 175/291 [00:44<00:23,  4.91it/s]
Loading 0:  60%|██████    | 176/291 [00:44<00:22,  5.15it/s]
Loading 0:  61%|██████    | 177/291 [00:44<00:19,  5.76it/s]
Loading 0:  62%|██████▏   | 179/291 [00:45<00:24,  4.60it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:32,  3.39it/s]
Loading 0:  62%|██████▏   | 181/291 [00:46<00:41,  2.66it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:23,  4.64it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:21,  4.88it/s]
Loading 0:  64%|██████▍   | 186/291 [00:46<00:19,  5.44it/s]
Loading 0:  64%|██████▍   | 187/291 [00:46<00:18,  5.53it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:28,  3.62it/s]
Loading 0:  65%|██████▍   | 189/291 [00:48<00:38,  2.68it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:27,  3.60it/s]
Loading 0:  66%|██████▋   | 193/291 [00:49<00:33,  2.95it/s]
Loading 0:  67%|██████▋   | 194/291 [00:50<00:40,  2.42it/s]
Loading 0:  68%|██████▊   | 197/291 [00:50<00:23,  4.07it/s]
Loading 0:  68%|██████▊   | 198/291 [00:50<00:21,  4.33it/s]
Loading 0:  68%|██████▊   | 199/291 [00:50<00:18,  4.88it/s]
Loading 0:  69%|██████▉   | 201/291 [00:51<00:21,  4.23it/s]
Loading 0:  69%|██████▉   | 202/291 [00:51<00:27,  3.25it/s]
Loading 0:  70%|██████▉   | 203/291 [00:52<00:33,  2.61it/s]
Loading 0:  71%|███████   | 206/291 [00:52<00:19,  4.41it/s]
Loading 0:  71%|███████   | 207/291 [00:52<00:17,  4.67it/s]
Loading 0:  71%|███████▏  | 208/291 [00:52<00:15,  5.23it/s]
Loading 0:  72%|███████▏  | 210/291 [00:53<00:18,  4.46it/s]
Loading 0:  73%|███████▎  | 211/291 [00:53<00:23,  3.36it/s]
Loading 0:  73%|███████▎  | 212/291 [00:54<00:29,  2.66it/s]
Loading 0:  74%|███████▍  | 215/291 [00:54<00:16,  4.56it/s]
Loading 0:  74%|███████▍  | 216/291 [00:54<00:15,  4.80it/s]
Loading 0:  75%|███████▍  | 217/291 [00:54<00:13,  5.33it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:16,  4.48it/s]
Loading 0:  76%|███████▌  | 220/291 [00:56<00:21,  3.37it/s]
Loading 0:  76%|███████▌  | 221/291 [00:56<00:26,  2.62it/s]
Loading 0:  77%|███████▋  | 224/291 [00:56<00:15,  4.45it/s]
Loading 0:  77%|███████▋  | 225/291 [00:57<00:14,  4.68it/s]
Loading 0:  78%|███████▊  | 226/291 [00:57<00:12,  5.21it/s]
Loading 0:  78%|███████▊  | 228/291 [00:57<00:14,  4.33it/s]
Loading 0:  79%|███████▊  | 229/291 [00:58<00:19,  3.26it/s]
Loading 0:  79%|███████▉  | 230/291 [00:59<00:23,  2.59it/s]
Loading 0:  80%|████████  | 233/291 [00:59<00:13,  4.39it/s]
Loading 0:  80%|████████  | 234/291 [00:59<00:12,  4.64it/s]
Loading 0:  81%|████████  | 235/291 [00:59<00:10,  5.14it/s]
Loading 0:  81%|████████▏ | 237/291 [01:00<00:12,  4.36it/s]
Loading 0:  82%|████████▏ | 238/291 [01:00<00:15,  3.33it/s]
Loading 0:  82%|████████▏ | 239/291 [01:01<00:19,  2.62it/s]
Loading 0:  83%|████████▎ | 242/291 [01:01<00:10,  4.52it/s]
Loading 0:  84%|████████▎ | 243/291 [01:01<00:10,  4.77it/s]
Loading 0:  84%|████████▍ | 244/291 [01:01<00:08,  5.33it/s]
Loading 0:  85%|████████▍ | 246/291 [01:02<00:10,  4.49it/s]
Loading 0:  85%|████████▍ | 247/291 [01:02<00:12,  3.40it/s]
Loading 0:  85%|████████▌ | 248/291 [01:03<00:16,  2.67it/s]
Loading 0:  86%|████████▋ | 251/291 [01:03<00:08,  4.60it/s]
Loading 0:  87%|████████▋ | 252/291 [01:03<00:08,  4.84it/s]
Loading 0:  87%|████████▋ | 253/291 [01:03<00:07,  5.38it/s]
Loading 0:  88%|████████▊ | 255/291 [01:04<00:08,  4.48it/s]
Loading 0:  88%|████████▊ | 256/291 [01:05<00:10,  3.34it/s]
Loading 0:  88%|████████▊ | 257/291 [01:05<00:12,  2.63it/s]
Loading 0:  89%|████████▉ | 260/291 [01:05<00:06,  4.51it/s]
Loading 0:  90%|████████▉ | 261/291 [01:06<00:06,  4.74it/s]
Loading 0:  90%|█████████ | 262/291 [01:06<00:05,  5.32it/s]
Loading 0:  91%|█████████ | 264/291 [01:06<00:06,  4.39it/s]
Loading 0:  91%|█████████ | 265/291 [01:07<00:07,  3.30it/s]
Loading 0:  91%|█████████▏| 266/291 [01:08<00:09,  2.61it/s]
Loading 0:  92%|█████████▏| 269/291 [01:08<00:04,  4.48it/s]
Loading 0:  93%|█████████▎| 270/291 [01:08<00:04,  4.72it/s]
Loading 0:  93%|█████████▎| 271/291 [01:08<00:03,  5.26it/s]
Loading 0:  94%|█████████▍| 273/291 [01:09<00:04,  4.44it/s]
Loading 0:  94%|█████████▍| 274/291 [01:09<00:05,  3.36it/s]
Loading 0:  95%|█████████▍| 275/291 [01:10<00:06,  2.66it/s]
Loading 0:  96%|█████████▌| 278/291 [01:10<00:02,  4.48it/s]
Loading 0:  96%|█████████▌| 279/291 [01:10<00:02,  4.70it/s]
Loading 0:  96%|█████████▌| 280/291 [01:10<00:02,  5.22it/s]
Loading 0:  97%|█████████▋| 281/291 [01:11<00:02,  3.57it/s]
Loading 0:  97%|█████████▋| 282/291 [01:11<00:03,  2.69it/s]
Loading 0:  98%|█████████▊| 284/291 [01:12<00:01,  3.85it/s]
Loading 0:  98%|█████████▊| 285/291 [01:12<00:01,  4.20it/s]
Loading 0:  98%|█████████▊| 286/291 [01:12<00:01,  4.85it/s]
Loading 0:  99%|█████████▊| 287/291 [01:12<00:00,  5.15it/s]
Loading 0:  99%|█████████▉| 288/291 [01:13<00:00,  3.33it/s]
                                                            
Job rirv938-20250402-reward-22740-v2-mkmlizer completed after 144.82s with status: succeeded
Stopping job with name rirv938-20250402-reward-22740-v2-mkmlizer
Pipeline stage MKMLizer completed in 145.29s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-20250402-reward-22740-v2
Waiting for inference service rirv938-20250402-reward-22740-v2 to be ready
Failed to get response for submission chaiml-sft-gemma2-28b-v_83370_v3: HTTPConnectionPool(host='chaiml-sft-gemma2-28b-v-83370-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service rirv938-20250402-reward-22740-v2 ready after 110.40919041633606s
Pipeline stage MKMLDeployer completed in 111.08s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.167144060134888s
Received healthy response to inference request in 4.88564658164978s
Received healthy response to inference request in 2.318934917449951s
Received healthy response to inference request in 2.366407871246338s
Received healthy response to inference request in 4.353520154953003s
5 requests
0 failed requests
5th percentile: 2.3284295082092283
10th percentile: 2.337924098968506
20th percentile: 2.3569132804870607
30th percentile: 2.763830327987671
40th percentile: 3.558675241470337
50th percentile: 4.353520154953003
60th percentile: 4.566370725631714
70th percentile: 4.779221296310425
80th percentile: 5.541946077346802
90th percentile: 6.854545068740845
95th percentile: 7.510844564437866
99th percentile: 8.035884160995483
mean time: 4.418330717086792
%s, retrying in %s seconds...
Received healthy response to inference request in 4.291219234466553s
Received healthy response to inference request in 3.1836740970611572s
Received healthy response to inference request in 2.3648200035095215s
Received healthy response to inference request in 2.4280853271484375s
Received healthy response to inference request in 3.873444080352783s
5 requests
0 failed requests
5th percentile: 2.377473068237305
10th percentile: 2.390126132965088
20th percentile: 2.415432262420654
30th percentile: 2.5792030811309816
40th percentile: 2.8814385890960694
50th percentile: 3.1836740970611572
60th percentile: 3.4595820903778076
70th percentile: 3.735490083694458
80th percentile: 3.9569991111755374
90th percentile: 4.124109172821045
95th percentile: 4.207664203643799
99th percentile: 4.274508228302002
mean time: 3.2282485485076906
%s, retrying in %s seconds...
Received healthy response to inference request in 5.443805932998657s
Received healthy response to inference request in 2.551805257797241s
Received healthy response to inference request in 2.0584683418273926s
Received healthy response to inference request in 1.9723584651947021s
Received healthy response to inference request in 2.4204537868499756s
5 requests
0 failed requests
5th percentile: 1.9895804405212403
10th percentile: 2.0068024158477784
20th percentile: 2.0412463665008547
30th percentile: 2.1308654308319093
40th percentile: 2.275659608840942
50th percentile: 2.4204537868499756
60th percentile: 2.4729943752288817
70th percentile: 2.525534963607788
80th percentile: 3.130205392837525
90th percentile: 4.287005662918091
95th percentile: 4.865405797958373
99th percentile: 5.328125905990601
mean time: 2.8893783569335936
Pipeline stage StressChecker completed in 56.84s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.74s
Shutdown handler de-registered
rirv938-20250402-reward_22740_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250402-reward-22740-v2-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.58s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250402-reward-22740-v2-profiler
Waiting for inference service rirv938-20250402-reward-22740-v2-profiler to be ready
Inference service rirv938-20250402-reward-22740-v2-profiler ready after 40.18740129470825s
Pipeline stage MKMLProfilerDeployer completed in 40.54s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250402-rewab6139b767fcd6b6aa3907b80499408a-deplo2mcnf:/code/chaiverse_profiler_1743650378 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250402-rewab6139b767fcd6b6aa3907b80499408a-deplo2mcnf --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1743650378 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1743650378/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250402-reward-22740-v2-profiler is running
Tearing down inference service rirv938-20250402-reward-22740-v2-profiler
Service rirv938-20250402-reward-22740-v2-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.57s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250402-reward-22740-v2-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.42s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250402-reward-22740-v2-profiler
Waiting for inference service rirv938-20250402-reward-22740-v2-profiler to be ready
Inference service rirv938-20250402-reward-22740-v2-profiler ready after 120.50002813339233s
Pipeline stage MKMLProfilerDeployer completed in 121.02s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250402-rewab6139b767fcd6b6aa3907b80499408a-deplorzx7k:/code/chaiverse_profiler_1743654085 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250402-rewab6139b767fcd6b6aa3907b80499408a-deplorzx7k --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1743654085 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1743654085/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250402-reward-22740-v2-profiler is running
Tearing down inference service rirv938-20250402-reward-22740-v2-profiler
Service rirv938-20250402-reward-22740-v2-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.48s
Shutdown handler de-registered
rirv938-20250402-reward_22740_v2 status is now inactive due to auto deactivation removed underperforming models
rirv938-20250402-reward_22740_v2 status is now torndown due to DeploymentManager action