chaiml-mn12b-syn1-tune24

developer_uid: ChaiHarshitSheoran
submission_id: chaiml-mn12b-syn1-tune24_v5
model_name: chaiml-mn12b-syn1-tune24_v5
model_group: ChaiML/mn12b_syn1_tune24
status: torndown
timestamp: 2025-09-06T14:23:01+00:00
num_battles: 5644
num_wins: 2878
celo_rating: 1253.74
family_friendly_score: 0.5636
family_friendly_standard_error: 0.007013630158484265
submission_type: basic
model_repo: ChaiML/mn12b_syn1_tune24
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.9308921408045029, 'latency_mean': 1.0741413748264312, 'latency_p50': 1.0742331743240356, 'latency_p90': 1.1975413322448731}, {'batch_size': 5, 'throughput': 2.4444819288350343, 'latency_mean': 2.0369380378723143, 'latency_p50': 2.031086564064026, 'latency_p90': 2.2611809730529786}, {'batch_size': 10, 'throughput': 3.028522232422747, 'latency_mean': 3.269114373922348, 'latency_p50': 3.2802740335464478, 'latency_p90': 3.6767906188964843}, {'batch_size': 15, 'throughput': 3.2622590663987037, 'latency_mean': 4.544986968040466, 'latency_p50': 4.528586506843567, 'latency_p90': 5.175854015350342}, {'batch_size': 20, 'throughput': 3.3232767655140565, 'latency_mean': 5.921713846921921, 'latency_p50': 5.921830773353577, 'latency_p90': 6.7677316904068}]
gpu_counts: {'NVIDIA L40S': 1}
display_name: chaiml-mn12b-syn1-tune24_v5
is_internal_developer: False
language_model: ChaiML/mn12b_syn1_tune24
model_size: 13B
ranking_group: single
throughput_3p7s: 3.16
us_pacific_date: 2025-09-06
win_ratio: 0.5099220411055989
generation_params: {'temperature': 0.9, 'top_p': 0.9, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-mn12b-syn1-tune24-v5-mkmlizer
Waiting for job on chaiml-mn12b-syn1-tune24-v5-mkmlizer to finish
HTTP Request: %s %s "%s %d %s"
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║                                                                     ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║                                                                     ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  Version: 0.30.2                                                    ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║                                                                     ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  belonging to:                                                      ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║                                                                     ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ║                                                                     ║
chaiml-mn12b-syn1-tune24-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
HTTP Request: %s %s "%s %d %s"
chaiml-mn12b-syn1-tune24-v5-mkmlizer: Downloaded to shared memory in 18.087s
chaiml-mn12b-syn1-tune24-v5-mkmlizer: Checking if ChaiML/mn12b_syn1_tune24 already exists in ChaiML
chaiml-mn12b-syn1-tune24-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmppfktfcp6, device:0
chaiml-mn12b-syn1-tune24-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-mn12b-syn1-tune24-v5-mkmlizer: quantized model in 147.944s
chaiml-mn12b-syn1-tune24-v5-mkmlizer: Processed model ChaiML/mn12b_syn1_tune24 in 166.031s
chaiml-mn12b-syn1-tune24-v5-mkmlizer: creating bucket guanaco-mkml-models
chaiml-mn12b-syn1-tune24-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-mn12b-syn1-tune24-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mn12b-syn1-tune24-v5/nvidia
chaiml-mn12b-syn1-tune24-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mn12b-syn1-tune24-v5/nvidia/special_tokens_map.json
chaiml-mn12b-syn1-tune24-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mn12b-syn1-tune24-v5/nvidia/config.json
chaiml-mn12b-syn1-tune24-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mn12b-syn1-tune24-v5/nvidia/tokenizer_config.json
chaiml-mn12b-syn1-tune24-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-mn12b-syn1-tune24-v5/nvidia/flywheel_model.0.safetensors
chaiml-mn12b-syn1-tune24-v5-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:00<01:46,  3.39it/s]
Loading 0:   1%|          | 4/363 [00:01<02:55,  2.05it/s]
Loading 0:   1%|▏         | 5/363 [00:02<03:49,  1.56it/s]
Loading 0:   2%|▏         | 8/363 [00:03<01:58,  2.98it/s]
Loading 0:   2%|▏         | 9/363 [00:03<01:54,  3.08it/s]
Loading 0:   3%|▎         | 10/363 [00:03<01:38,  3.59it/s]
Loading 0:   3%|▎         | 12/363 [00:04<02:00,  2.91it/s]
Loading 0:   4%|▎         | 13/363 [00:05<02:41,  2.17it/s]
Loading 0:   4%|▍         | 14/363 [00:06<03:22,  1.72it/s]
Loading 0:   5%|▍         | 17/363 [00:06<01:57,  2.95it/s]
Loading 0:   5%|▍         | 18/363 [00:06<01:51,  3.11it/s]
Loading 0:   5%|▌         | 19/363 [00:06<01:36,  3.58it/s]
Loading 0:   6%|▌         | 21/363 [00:07<01:56,  2.93it/s]
Loading 0:   6%|▌         | 22/363 [00:08<02:35,  2.19it/s]
Loading 0:   6%|▋         | 23/363 [00:09<03:15,  1.74it/s]
Loading 0:   7%|▋         | 26/363 [00:09<01:54,  2.95it/s]
Loading 0:   7%|▋         | 27/363 [00:10<01:48,  3.11it/s]
Loading 0:   8%|▊         | 28/363 [00:10<01:33,  3.59it/s]
Loading 0:   8%|▊         | 30/363 [00:11<01:53,  2.94it/s]
Loading 0:   9%|▊         | 31/363 [00:12<02:30,  2.20it/s]
Loading 0:   9%|▉         | 32/363 [00:13<03:10,  1.74it/s]
Loading 0:  10%|▉         | 35/363 [00:13<01:51,  2.95it/s]
Loading 0:  10%|▉         | 36/363 [00:13<01:45,  3.10it/s]
Loading 0:  10%|█         | 37/363 [00:13<01:30,  3.58it/s]
Loading 0:  11%|█         | 39/363 [00:14<01:50,  2.94it/s]
Loading 0:  11%|█         | 40/363 [00:15<02:26,  2.20it/s]
Loading 0:  11%|█▏        | 41/363 [00:16<03:04,  1.74it/s]
Loading 0:  12%|█▏        | 44/363 [00:16<01:47,  2.96it/s]
Loading 0:  12%|█▏        | 45/363 [00:16<01:42,  3.11it/s]
Loading 0:  13%|█▎        | 46/363 [00:17<01:27,  3.62it/s]
Loading 0:  13%|█▎        | 48/363 [00:17<01:46,  2.96it/s]
Loading 0:  13%|█▎        | 49/363 [00:18<02:22,  2.21it/s]
Loading 0:  14%|█▍        | 50/363 [00:19<02:57,  1.76it/s]
Loading 0:  15%|█▍        | 53/363 [00:20<01:45,  2.94it/s]
Loading 0:  15%|█▍        | 54/363 [00:20<01:40,  3.08it/s]
Loading 0:  15%|█▌        | 55/363 [00:20<01:27,  3.51it/s]
Loading 0:  15%|█▌        | 56/363 [00:21<02:12,  2.32it/s]
Loading 0:  16%|█▌        | 58/363 [00:21<01:37,  3.14it/s]
Loading 0:  16%|█▋        | 59/363 [00:21<01:32,  3.29it/s]
Loading 0:  17%|█▋        | 60/363 [00:22<01:19,  3.80it/s]
Loading 0:  17%|█▋        | 61/363 [00:22<01:11,  4.20it/s]
Loading 0:  17%|█▋        | 62/363 [00:23<02:03,  2.43it/s]
Loading 0:  17%|█▋        | 63/363 [00:24<02:42,  1.85it/s]
Loading 0:  18%|█▊        | 64/363 [00:24<03:15,  1.53it/s]
Loading 0:  18%|█▊        | 67/363 [00:25<01:44,  2.84it/s]
Loading 0:  19%|█▊        | 68/363 [00:25<01:37,  3.02it/s]
Loading 0:  19%|█▉        | 69/363 [00:25<01:23,  3.52it/s]
Loading 0:  20%|█▉        | 71/363 [00:26<01:43,  2.81it/s]
Loading 0:  20%|█▉        | 72/363 [00:27<02:28,  1.96it/s]
Loading 0:  20%|██        | 73/363 [00:28<03:01,  1.60it/s]
Loading 0:  21%|██        | 76/363 [00:28<01:43,  2.76it/s]
Loading 0:  21%|██        | 77/363 [00:29<01:37,  2.94it/s]
Loading 0:  21%|██▏       | 78/363 [00:29<01:23,  3.43it/s]
Loading 0:  22%|██▏       | 80/363 [00:30<01:38,  2.86it/s]
Loading 0:  22%|██▏       | 81/363 [00:31<02:10,  2.16it/s]
Loading 0:  23%|██▎       | 82/363 [00:32<02:42,  1.73it/s]
Loading 0:  23%|██▎       | 85/363 [00:32<01:34,  2.95it/s]
Loading 0:  24%|██▎       | 86/363 [00:32<01:29,  3.10it/s]
Loading 0:  24%|██▍       | 87/363 [00:32<01:16,  3.59it/s]
Loading 0:  25%|██▍       | 89/363 [00:33<01:33,  2.93it/s]
Loading 0:  25%|██▍       | 90/363 [00:34<02:04,  2.19it/s]
Loading 0:  25%|██▌       | 91/363 [00:35<02:36,  1.74it/s]
Loading 0:  26%|██▌       | 94/363 [00:35<01:31,  2.95it/s]
Loading 0:  26%|██▌       | 95/363 [00:36<01:26,  3.10it/s]
Loading 0:  26%|██▋       | 96/363 [00:36<01:14,  3.58it/s]
Loading 0:  27%|██▋       | 98/363 [00:37<01:30,  2.93it/s]
Loading 0:  27%|██▋       | 99/363 [00:37<02:00,  2.19it/s]
Loading 0:  28%|██▊       | 100/363 [00:38<02:30,  1.75it/s]
Loading 0:  28%|██▊       | 103/363 [00:39<01:28,  2.95it/s]
Loading 0:  29%|██▊       | 104/363 [00:39<01:23,  3.10it/s]
Loading 0:  29%|██▉       | 105/363 [00:39<01:12,  3.57it/s]
Loading 0:  29%|██▉       | 106/363 [00:40<01:48,  2.37it/s]
Loading 0:  30%|██▉       | 108/363 [00:40<01:20,  3.18it/s]
Loading 0:  30%|███       | 109/363 [00:41<01:16,  3.32it/s]
Loading 0:  30%|███       | 110/363 [00:41<01:05,  3.89it/s]
Loading 0:  31%|███       | 112/363 [00:42<01:23,  3.01it/s]
Loading 0:  31%|███       | 113/363 [00:42<01:58,  2.11it/s]
Loading 0:  32%|███▏      | 116/363 [00:43<01:35,  2.60it/s]
Loading 0:  32%|███▏      | 117/363 [00:44<01:57,  2.09it/s]
Loading 0:  33%|███▎      | 118/363 [00:45<02:21,  1.73it/s]
Loading 0:  33%|███▎      | 121/363 [00:45<01:25,  2.84it/s]
Loading 0:  34%|███▎      | 122/363 [00:46<01:20,  3.00it/s]
Loading 0:  34%|███▍      | 123/363 [00:46<01:09,  3.46it/s]
Loading 0:  34%|███▍      | 125/363 [00:47<01:22,  2.89it/s]
Loading 0:  35%|███▍      | 126/363 [00:48<01:48,  2.19it/s]
Loading 0:  35%|███▍      | 127/363 [00:49<02:14,  1.75it/s]
Loading 0:  36%|███▌      | 130/363 [00:49<01:18,  2.95it/s]
Loading 0:  36%|███▌      | 131/363 [00:49<01:14,  3.10it/s]
Loading 0:  36%|███▋      | 132/363 [00:49<01:04,  3.59it/s]
Loading 0:  37%|███▋      | 134/363 [00:50<01:18,  2.93it/s]
Loading 0:  37%|███▋      | 135/363 [00:51<01:43,  2.20it/s]
Loading 0:  37%|███▋      | 136/363 [00:52<02:09,  1.75it/s]
Loading 0:  38%|███▊      | 139/363 [00:52<01:15,  2.96it/s]
Loading 0:  39%|███▊      | 140/363 [00:53<01:11,  3.11it/s]
Loading 0:  39%|███▉      | 141/363 [00:53<01:02,  3.57it/s]
Loading 0:  39%|███▉      | 142/363 [00:53<00:55,  3.99it/s]
Loading 0:  39%|███▉      | 143/363 [00:54<01:30,  2.44it/s]
Loading 0:  40%|███▉      | 144/363 [00:55<02:00,  1.81it/s]
Loading 0:  40%|████      | 147/363 [00:56<01:29,  2.41it/s]
Loading 0:  41%|████      | 148/363 [00:56<01:49,  1.96it/s]
Loading 0:  41%|████      | 149/363 [00:57<02:09,  1.65it/s]
Loading 0:  42%|████▏     | 152/363 [00:58<01:16,  2.77it/s]
Loading 0:  42%|████▏     | 153/363 [00:58<01:11,  2.94it/s]
Loading 0:  42%|████▏     | 154/363 [00:58<01:01,  3.39it/s]
Loading 0:  43%|████▎     | 156/363 [00:59<01:12,  2.85it/s]
Loading 0:  43%|████▎     | 157/363 [01:00<01:35,  2.16it/s]
Loading 0:  44%|████▎     | 158/363 [01:01<01:58,  1.73it/s]
Loading 0:  44%|████▍     | 161/363 [01:01<01:08,  2.93it/s]
Loading 0:  45%|████▍     | 162/363 [01:01<01:05,  3.09it/s]
Loading 0:  45%|████▍     | 163/363 [01:01<00:56,  3.56it/s]
Loading 0:  45%|████▌     | 165/363 [01:02<01:07,  2.92it/s]
Loading 0:  46%|████▌     | 166/363 [01:03<01:30,  2.18it/s]
Loading 0:  46%|████▌     | 167/363 [01:04<01:52,  1.75it/s]
Loading 0:  47%|████▋     | 170/363 [01:04<01:05,  2.96it/s]
Loading 0:  47%|████▋     | 171/363 [01:05<01:01,  3.11it/s]
Loading 0:  47%|████▋     | 172/363 [01:05<00:53,  3.57it/s]
Loading 0:  48%|████▊     | 174/363 [01:06<01:04,  2.93it/s]
Loading 0:  48%|████▊     | 175/363 [01:07<01:25,  2.20it/s]
Loading 0:  48%|████▊     | 176/363 [01:08<01:46,  1.75it/s]
Loading 0:  49%|████▉     | 179/363 [01:08<01:02,  2.96it/s]
Loading 0:  50%|████▉     | 180/363 [01:08<00:58,  3.11it/s]
Loading 0:  50%|████▉     | 181/363 [01:08<00:50,  3.60it/s]
Loading 0:  50%|█████     | 183/363 [01:09<01:01,  2.95it/s]
Loading 0:  51%|█████     | 184/363 [01:10<01:21,  2.21it/s]
Loading 0:  51%|█████     | 185/363 [01:11<01:40,  1.77it/s]
Loading 0:  52%|█████▏    | 188/363 [01:11<00:58,  2.99it/s]
Loading 0:  52%|█████▏    | 189/363 [01:12<00:55,  3.14it/s]
Loading 0:  52%|█████▏    | 190/363 [01:12<00:47,  3.62it/s]
Loading 0:  53%|█████▎    | 192/363 [01:13<00:57,  2.96it/s]
Loading 0:  53%|█████▎    | 193/363 [01:13<01:17,  2.21it/s]
Loading 0:  53%|█████▎    | 194/363 [01:14<01:36,  1.76it/s]
Loading 0:  54%|█████▍    | 197/363 [01:15<00:55,  2.98it/s]
Loading 0:  55%|█████▍    | 198/363 [01:15<00:52,  3.13it/s]
Loading 0:  55%|█████▍    | 199/363 [01:15<00:45,  3.61it/s]
Loading 0:  55%|█████▌    | 201/363 [01:16<00:54,  2.95it/s]
Loading 0:  56%|█████▌    | 202/363 [01:17<01:12,  2.21it/s]
Loading 0:  56%|█████▌    | 203/363 [01:18<01:30,  1.77it/s]
Loading 0:  57%|█████▋    | 206/363 [01:18<00:52,  2.99it/s]
Loading 0:  57%|█████▋    | 207/363 [01:18<00:49,  3.14it/s]
Loading 0:  57%|█████▋    | 208/363 [01:18<00:42,  3.62it/s]
Loading 0:  58%|█████▊    | 210/363 [01:19<00:51,  2.96it/s]
Loading 0:  58%|█████▊    | 211/363 [01:20<01:08,  2.21it/s]
Loading 0:  58%|█████▊    | 212/363 [01:21<01:25,  1.77it/s]
Loading 0:  59%|█████▉    | 215/363 [01:21<00:49,  2.99it/s]
Loading 0:  60%|█████▉    | 216/363 [01:22<00:46,  3.15it/s]
Loading 0:  60%|█████▉    | 217/363 [01:22<00:40,  3.59it/s]
Loading 0:  60%|██████    | 218/363 [01:23<01:00,  2.39it/s]
Loading 0:  61%|██████    | 220/363 [01:23<00:44,  3.20it/s]
Loading 0:  61%|██████    | 221/363 [01:23<00:42,  3.35it/s]
Loading 0:  61%|██████    | 222/363 [01:23<00:36,  3.91it/s]
Loading 0:  61%|██████▏   | 223/363 [01:23<00:31,  4.42it/s]
Loading 0:  62%|██████▏   | 224/363 [01:24<00:55,  2.50it/s]
Loading 0:  62%|██████▏   | 225/363 [01:25<01:15,  1.82it/s]
Loading 0:  63%|██████▎   | 228/363 [01:26<00:55,  2.44it/s]
Loading 0:  63%|██████▎   | 229/363 [01:27<01:07,  1.98it/s]
Loading 0:  63%|██████▎   | 230/363 [01:28<01:19,  1.66it/s]
Loading 0:  64%|██████▍   | 233/363 [01:28<00:46,  2.81it/s]
Loading 0:  64%|██████▍   | 234/363 [01:29<00:43,  2.98it/s]
Loading 0:  65%|██████▍   | 235/363 [01:29<00:37,  3.45it/s]
Loading 0:  65%|██████▌   | 237/363 [01:30<00:43,  2.89it/s]
Loading 0:  66%|██████▌   | 238/363 [01:30<00:57,  2.19it/s]
Loading 0:  66%|██████▌   | 239/363 [01:31<01:10,  1.75it/s]
Loading 0:  67%|██████▋   | 242/363 [01:32<00:40,  2.97it/s]
Loading 0:  67%|██████▋   | 243/363 [01:32<00:38,  3.12it/s]
Loading 0:  67%|██████▋   | 244/363 [01:32<00:33,  3.60it/s]
Loading 0:  68%|██████▊   | 246/363 [01:33<00:39,  2.96it/s]
Loading 0:  68%|██████▊   | 247/363 [01:34<00:52,  2.21it/s]
Loading 0:  68%|██████▊   | 248/363 [01:35<01:04,  1.78it/s]
Loading 0:  69%|██████▉   | 251/363 [01:35<00:37,  3.01it/s]
Loading 0:  69%|██████▉   | 252/363 [01:35<00:35,  3.16it/s]
Loading 0:  70%|██████▉   | 253/363 [01:35<00:30,  3.62it/s]
Loading 0:  70%|███████   | 255/363 [01:36<00:36,  2.97it/s]
Loading 0:  71%|███████   | 256/363 [01:37<00:48,  2.22it/s]
Loading 0:  71%|███████   | 257/363 [01:38<00:59,  1.77it/s]
Loading 0:  72%|███████▏  | 260/363 [01:38<00:34,  3.00it/s]
Loading 0:  72%|███████▏  | 261/363 [01:39<00:32,  3.15it/s]
Loading 0:  72%|███████▏  | 262/363 [01:39<00:27,  3.64it/s]
Loading 0:  73%|███████▎  | 264/363 [01:40<00:33,  2.97it/s]
Loading 0:  73%|███████▎  | 265/363 [01:40<00:44,  2.22it/s]
Loading 0:  73%|███████▎  | 266/363 [01:41<00:54,  1.78it/s]
Loading 0:  74%|███████▍  | 269/363 [01:42<00:31,  3.00it/s]
Loading 0:  74%|███████▍  | 270/363 [01:42<00:29,  3.14it/s]
Loading 0:  75%|███████▍  | 271/363 [01:42<00:25,  3.58it/s]
Loading 0:  75%|███████▌  | 273/363 [01:43<00:30,  2.95it/s]
Loading 0:  75%|███████▌  | 274/363 [01:44<00:40,  2.21it/s]
Loading 0:  76%|███████▌  | 275/363 [01:45<00:49,  1.77it/s]
Loading 0:  77%|███████▋  | 278/363 [01:45<00:28,  3.00it/s]
Loading 0:  77%|███████▋  | 279/363 [01:45<00:26,  3.14it/s]
Loading 0:  77%|███████▋  | 280/363 [01:45<00:23,  3.61it/s]
Loading 0:  78%|███████▊  | 282/363 [01:46<00:27,  2.96it/s]
Loading 0:  78%|███████▊  | 283/363 [01:47<00:36,  2.21it/s]
Loading 0:  78%|███████▊  | 284/363 [01:48<00:44,  1.78it/s]
Loading 0:  79%|███████▉  | 287/363 [01:48<00:25,  3.00it/s]
Loading 0:  79%|███████▉  | 288/363 [01:49<00:23,  3.15it/s]
Loading 0:  80%|███████▉  | 289/363 [01:49<00:20,  3.63it/s]
Loading 0:  80%|████████  | 291/363 [01:50<00:24,  2.97it/s]
Loading 0:  80%|████████  | 292/363 [01:51<00:32,  2.22it/s]
Loading 0:  81%|████████  | 293/363 [01:51<00:39,  1.77it/s]
Loading 0:  82%|████████▏ | 296/363 [01:52<00:22,  2.99it/s]
Loading 0:  82%|████████▏ | 297/363 [01:52<00:21,  3.14it/s]
Loading 0:  82%|████████▏ | 298/363 [01:52<00:17,  3.62it/s]
Loading 0:  82%|████████▏ | 299/363 [01:53<00:26,  2.40it/s]
Loading 0:  83%|████████▎ | 301/363 [01:53<00:19,  3.21it/s]
Loading 0:  83%|████████▎ | 302/363 [01:54<00:18,  3.35it/s]
Loading 0:  83%|████████▎ | 303/363 [01:54<00:15,  3.92it/s]
Loading 0:  84%|████████▎ | 304/363 [01:54<00:15,  3.74it/s]
Loading 0:  84%|████████▍ | 306/363 [01:55<00:19,  2.95it/s]
Loading 0:  85%|████████▍ | 307/363 [01:56<00:26,  2.10it/s]
Loading 0:  85%|████████▌ | 310/363 [01:57<00:20,  2.61it/s]
Loading 0:  86%|████████▌ | 311/363 [01:58<00:24,  2.10it/s]
Loading 0:  86%|████████▌ | 312/363 [01:59<00:29,  1.74it/s]
Loading 0:  87%|████████▋ | 315/363 [01:59<00:16,  2.86it/s]
Loading 0:  87%|████████▋ | 316/363 [01:59<00:15,  3.02it/s]
Loading 0:  87%|████████▋ | 317/363 [01:59<00:13,  3.48it/s]
Loading 0:  88%|████████▊ | 319/363 [02:00<00:15,  2.92it/s]
Loading 0:  88%|████████▊ | 320/363 [02:01<00:19,  2.20it/s]
Loading 0:  88%|████████▊ | 321/363 [02:02<00:23,  1.76it/s]
Loading 0:  89%|████████▉ | 324/363 [02:02<00:13,  2.98it/s]
Loading 0:  90%|████████▉ | 325/363 [02:02<00:12,  3.12it/s]
Loading 0:  90%|████████▉ | 326/363 [02:03<00:10,  3.61it/s]
Loading 0:  90%|█████████ | 328/363 [02:03<00:11,  2.96it/s]
Loading 0:  91%|█████████ | 329/363 [02:04<00:15,  2.22it/s]
Loading 0:  91%|█████████ | 330/363 [02:05<00:18,  1.76it/s]
Loading 0:  92%|█████████▏| 333/363 [02:06<00:10,  2.98it/s]
Loading 0:  92%|█████████▏| 334/363 [02:06<00:09,  3.13it/s]
Loading 0:  92%|█████████▏| 335/363 [02:06<00:07,  3.61it/s]
Loading 0:  93%|█████████▎| 337/363 [02:07<00:08,  2.96it/s]
Loading 0:  93%|█████████▎| 338/363 [02:08<00:11,  2.21it/s]
Loading 0:  93%|█████████▎| 339/363 [02:09<00:13,  1.77it/s]
Loading 0:  94%|█████████▍| 342/363 [02:09<00:07,  3.00it/s]
Loading 0:  94%|█████████▍| 343/363 [02:09<00:06,  3.14it/s]
Loading 0:  95%|█████████▍| 344/363 [02:09<00:05,  3.62it/s]
Loading 0:  95%|█████████▌| 346/363 [02:10<00:05,  2.96it/s]
Loading 0:  96%|█████████▌| 347/363 [02:11<00:07,  2.21it/s]
Loading 0:  96%|█████████▌| 348/363 [02:12<00:08,  1.77it/s]
Loading 0:  97%|█████████▋| 351/363 [02:12<00:04,  2.99it/s]
Loading 0:  97%|█████████▋| 352/363 [02:13<00:03,  3.14it/s]
Loading 0:  97%|█████████▋| 353/363 [02:13<00:02,  3.59it/s]
Loading 0:  98%|█████████▊| 355/363 [02:14<00:02,  2.91it/s]
Loading 0:  98%|█████████▊| 356/363 [02:14<00:03,  2.19it/s]
Loading 0:  98%|█████████▊| 357/363 [02:15<00:03,  1.76it/s]
Loading 0:  99%|█████████▉| 360/363 [02:16<00:01,  2.97it/s]
Loading 0:  99%|█████████▉| 361/363 [02:16<00:00,  3.13it/s]
Loading 0: 100%|█████████▉| 362/363 [02:16<00:00,  3.60it/s]
                                                            
Job chaiml-mn12b-syn1-tune24-v5-mkmlizer completed after 254.5s with status: succeeded
Stopping job with name chaiml-mn12b-syn1-tune24-v5-mkmlizer
Pipeline stage MKMLizer completed in 256.01s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 1.20s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mn12b-syn1-tune24-v5
Waiting for inference service chaiml-mn12b-syn1-tune24-v5 to be ready
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-mn12b-syn1-tune24-v5 ready after 70.40565943717957s
Pipeline stage MKMLDeployer completed in 72.29s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.3370835781097412s
Received healthy response to inference request in 1.4857971668243408s
Received healthy response to inference request in 1.924912452697754s
Received healthy response to inference request in 1.3280673027038574s
Received healthy response to inference request in 6.39453911781311s
5 requests
0 failed requests
5th percentile: 1.3298705577850343
10th percentile: 1.331673812866211
20th percentile: 1.3352803230285644
30th percentile: 1.3668262958526611
40th percentile: 1.426311731338501
50th percentile: 1.4857971668243408
60th percentile: 1.661443281173706
70th percentile: 1.8370893955230712
80th percentile: 2.818837785720826
90th percentile: 4.606688451766968
95th percentile: 5.500613784790039
Failed to get response for submission blend_rurun_2025-08-28: ('http://guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com/models/chaiml-grpo-nis-12bsftj_56492_v4/predict', '{"detail":"list index out of range"}')
99th percentile: 6.215754051208496
mean time: 2.494079923629761
Pipeline stage StressChecker completed in 19.09s
HTTP Request: %s %s "%s %d %s"
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.91s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.79s
Shutdown handler de-registered
chaiml-mn12b-syn1-tune24_v5 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2380.58s
Shutdown handler de-registered
chaiml-mn12b-syn1-tune24_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-mn12b-syn1-tune24_v5 status is now torndown due to DeploymentManager action