chaiml-grpo-nis-12bsft_56492

developer_uid: zonemercy
submission_id: chaiml-grpo-nis-12bsft_56492_v15
model_name: chaiml-grpo-nis-12bsft_56492_v15
model_group: ChaiML/grpo_nis_12bsftju
status: torndown
timestamp: 2025-10-02T22:33:53+00:00
num_battles: 5404
num_wins: 2600
celo_rating: 1273.24
family_friendly_score: 0.5247999999999999
family_friendly_standard_error: 0.007062364476575816
submission_type: basic
model_repo: ChaiML/grpo_nis_12bsftjun_v4_cp_beta30_rm1_s1_step1370_e10_merged
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 7
max_input_tokens: 1280
max_output_tokens: 60
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.9363388786276043, 'latency_mean': 1.0678664124011994, 'latency_p50': 1.0693949460983276, 'latency_p90': 1.2001937866210937}, {'batch_size': 5, 'throughput': 2.380078095177588, 'latency_mean': 2.0894122183322907, 'latency_p50': 2.097098708152771, 'latency_p90': 2.339839720726013}, {'batch_size': 10, 'throughput': 2.873886766920657, 'latency_mean': 3.4486576604843138, 'latency_p50': 3.4743008613586426, 'latency_p90': 3.9040401935577393}, {'batch_size': 15, 'throughput': 3.0396778975047143, 'latency_mean': 4.879185475111008, 'latency_p50': 4.943344593048096, 'latency_p90': 5.59814567565918}, {'batch_size': 20, 'throughput': 3.1369266464847914, 'latency_mean': 6.274353017807007, 'latency_p50': 6.267553687095642, 'latency_p90': 7.203665995597839}]
gpu_counts: {'NVIDIA L40S': 1}
display_name: chaiml-grpo-nis-12bsft_56492_v15
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo_nis_12bsftjun_v4_cp_beta30_rm1_s1_step1370_e10_merged
model_size: 13B
ranking_group: single
throughput_3p7s: 2.94
us_pacific_date: 2025-10-02
win_ratio: 0.4811250925240563
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.15, 'frequency_penalty': 0.15, 'stopping_words': ['</s>', '###', '\n', 'You:'], 'max_input_tokens': 1280, 'best_of': 7, 'max_output_tokens': 60}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-grpo-nis-12bsft-56492-v15-mkmlizer
Waiting for job on chaiml-grpo-nis-12bsft-56492-v15-mkmlizer to finish
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║                                                                     ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║                                                                     ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  Version: 0.30.2                                                    ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║                                                                     ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  belonging to:                                                      ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║                                                                     ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ║                                                                     ║
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: Downloaded to shared memory in 27.488s
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: Checking if ChaiML/grpo_nis_12bsftjun_v4_cp_beta30_rm1_s1_step1370_e10_merged already exists in ChaiML
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpop4zd57n, device:0
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission mistralai-mistral-nem_93303_v569: ('http://mistralai-mistral-nem-93303-v569-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: quantized model in 145.508s
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: Processed model ChaiML/grpo_nis_12bsftjun_v4_cp_beta30_rm1_s1_step1370_e10_merged in 172.997s
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: creating bucket guanaco-mkml-models
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-grpo-nis-12bsft-56492-v15/nvidia
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-grpo-nis-12bsft-56492-v15/nvidia/config.json
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-grpo-nis-12bsft-56492-v15/nvidia/special_tokens_map.json
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-grpo-nis-12bsft-56492-v15/nvidia/tokenizer_config.json
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-grpo-nis-12bsft-56492-v15/nvidia/tokenizer.json
chaiml-grpo-nis-12bsft-56492-v15-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:00<01:43,  3.48it/s]
Loading 0:   1%|          | 4/363 [00:01<02:50,  2.11it/s]
Loading 0:   1%|▏         | 5/363 [00:02<03:44,  1.60it/s]
Loading 0:   2%|▏         | 8/363 [00:02<01:54,  3.10it/s]
Loading 0:   2%|▏         | 9/363 [00:03<01:50,  3.20it/s]
Loading 0:   3%|▎         | 10/363 [00:03<01:34,  3.72it/s]
Loading 0:   3%|▎         | 12/363 [00:04<02:05,  2.79it/s]
Loading 0:   4%|▎         | 13/363 [00:05<02:45,  2.12it/s]
Loading 0:   4%|▍         | 14/363 [00:06<03:24,  1.71it/s]
Loading 0:   5%|▍         | 17/363 [00:06<01:57,  2.95it/s]
Loading 0:   5%|▍         | 18/363 [00:06<01:50,  3.11it/s]
Loading 0:   5%|▌         | 19/363 [00:06<01:35,  3.62it/s]
Loading 0:   6%|▌         | 21/363 [00:07<01:54,  2.98it/s]
Loading 0:   6%|▌         | 22/363 [00:08<02:32,  2.24it/s]
Loading 0:   6%|▋         | 23/363 [00:09<03:10,  1.78it/s]
Loading 0:   7%|▋         | 26/363 [00:09<01:50,  3.06it/s]
Loading 0:   7%|▋         | 27/363 [00:10<01:44,  3.21it/s]
Loading 0:   8%|▊         | 28/363 [00:10<01:30,  3.72it/s]
Loading 0:   8%|▊         | 30/363 [00:10<01:49,  3.03it/s]
Loading 0:   9%|▊         | 31/363 [00:11<02:26,  2.26it/s]
Loading 0:   9%|▉         | 32/363 [00:12<03:03,  1.80it/s]
Loading 0:  10%|▉         | 35/363 [00:13<01:47,  3.05it/s]
Loading 0:  10%|▉         | 36/363 [00:13<01:42,  3.20it/s]
Loading 0:  10%|█         | 37/363 [00:13<01:28,  3.70it/s]
Loading 0:  11%|█         | 39/363 [00:14<01:47,  3.02it/s]
Loading 0:  11%|█         | 40/363 [00:15<02:23,  2.25it/s]
Loading 0:  11%|█▏        | 41/363 [00:16<02:59,  1.79it/s]
Loading 0:  12%|█▏        | 44/363 [00:16<01:45,  3.03it/s]
Loading 0:  12%|█▏        | 45/363 [00:16<01:39,  3.19it/s]
Loading 0:  13%|█▎        | 46/363 [00:16<01:25,  3.69it/s]
Loading 0:  13%|█▎        | 48/363 [00:17<01:44,  3.01it/s]
Loading 0:  13%|█▎        | 49/363 [00:18<02:19,  2.25it/s]
Loading 0:  14%|█▍        | 50/363 [00:19<02:54,  1.79it/s]
Loading 0:  15%|█▍        | 53/363 [00:19<01:42,  3.03it/s]
Loading 0:  15%|█▍        | 54/363 [00:19<01:37,  3.18it/s]
Loading 0:  15%|█▌        | 55/363 [00:20<01:25,  3.62it/s]
Loading 0:  15%|█▌        | 56/363 [00:20<02:06,  2.42it/s]
Loading 0:  16%|█▌        | 58/363 [00:21<01:33,  3.25it/s]
Loading 0:  16%|█▋        | 59/363 [00:21<01:29,  3.40it/s]
Loading 0:  17%|█▋        | 60/363 [00:21<01:16,  3.97it/s]
Loading 0:  17%|█▋        | 61/363 [00:21<01:08,  4.43it/s]
Loading 0:  17%|█▋        | 62/363 [00:22<01:59,  2.51it/s]
Loading 0:  17%|█▋        | 63/363 [00:23<02:37,  1.90it/s]
Loading 0:  18%|█▊        | 64/363 [00:24<03:11,  1.57it/s]
Loading 0:  18%|█▊        | 67/363 [00:24<01:42,  2.90it/s]
Loading 0:  19%|█▊        | 68/363 [00:24<01:35,  3.08it/s]
Loading 0:  19%|█▉        | 69/363 [00:25<01:22,  3.57it/s]
Loading 0:  20%|█▉        | 71/363 [00:25<01:39,  2.94it/s]
Loading 0:  20%|█▉        | 72/363 [00:26<02:12,  2.20it/s]
Loading 0:  20%|██        | 73/363 [00:27<02:45,  1.75it/s]
Loading 0:  21%|██        | 76/363 [00:28<01:36,  2.99it/s]
Loading 0:  21%|██        | 77/363 [00:28<01:30,  3.15it/s]
Loading 0:  21%|██▏       | 78/363 [00:28<01:18,  3.64it/s]
Loading 0:  22%|██▏       | 80/363 [00:29<01:34,  2.98it/s]
Loading 0:  22%|██▏       | 81/363 [00:30<02:07,  2.22it/s]
Loading 0:  23%|██▎       | 82/363 [00:31<02:38,  1.78it/s]
Loading 0:  23%|██▎       | 85/363 [00:31<01:32,  3.01it/s]
Loading 0:  24%|██▎       | 86/363 [00:31<01:27,  3.17it/s]
Loading 0:  24%|██▍       | 87/363 [00:31<01:15,  3.65it/s]
Loading 0:  25%|██▍       | 89/363 [00:32<01:31,  2.99it/s]
Loading 0:  25%|██▍       | 90/363 [00:33<02:02,  2.24it/s]
Loading 0:  25%|██▌       | 91/363 [00:34<02:33,  1.78it/s]
Loading 0:  26%|██▌       | 94/363 [00:34<01:29,  3.01it/s]
Loading 0:  26%|██▌       | 95/363 [00:35<01:24,  3.17it/s]
Loading 0:  26%|██▋       | 96/363 [00:35<01:14,  3.58it/s]
Loading 0:  27%|██▋       | 98/363 [00:36<01:29,  2.97it/s]
Loading 0:  27%|██▋       | 99/363 [00:36<01:58,  2.22it/s]
Loading 0:  28%|██▊       | 100/363 [00:37<02:27,  1.78it/s]
Loading 0:  28%|██▊       | 103/363 [00:38<01:26,  3.02it/s]
Loading 0:  29%|██▊       | 104/363 [00:38<01:21,  3.17it/s]
Loading 0:  29%|██▉       | 105/363 [00:38<01:10,  3.65it/s]
Loading 0:  29%|██▉       | 106/363 [00:39<01:45,  2.42it/s]
Loading 0:  30%|██▉       | 108/363 [00:39<01:18,  3.24it/s]
Loading 0:  30%|███       | 109/363 [00:39<01:14,  3.39it/s]
Loading 0:  30%|███       | 110/363 [00:40<01:03,  3.96it/s]
Loading 0:  31%|███       | 112/363 [00:40<01:21,  3.07it/s]
Loading 0:  31%|███       | 113/363 [00:41<01:55,  2.16it/s]
Loading 0:  32%|███▏      | 116/363 [00:42<01:32,  2.66it/s]
Loading 0:  32%|███▏      | 117/363 [00:43<01:54,  2.14it/s]
Loading 0:  33%|███▎      | 118/363 [00:44<02:18,  1.78it/s]
Loading 0:  33%|███▎      | 121/363 [00:44<01:23,  2.91it/s]
Loading 0:  34%|███▎      | 122/363 [00:45<01:18,  3.08it/s]
Loading 0:  34%|███▍      | 123/363 [00:45<01:07,  3.55it/s]
Loading 0:  34%|███▍      | 125/363 [00:46<01:20,  2.97it/s]
Loading 0:  35%|███▍      | 126/363 [00:46<01:45,  2.24it/s]
Loading 0:  35%|███▍      | 127/363 [00:47<02:10,  1.80it/s]
Loading 0:  36%|███▌      | 130/363 [00:48<01:16,  3.05it/s]
Loading 0:  36%|███▌      | 131/363 [00:48<01:12,  3.20it/s]
Loading 0:  36%|███▋      | 132/363 [00:48<01:02,  3.70it/s]
Loading 0:  37%|███▋      | 134/363 [00:49<01:16,  3.01it/s]
Loading 0:  37%|███▋      | 135/363 [00:50<01:41,  2.25it/s]
Loading 0:  37%|███▋      | 136/363 [00:51<02:06,  1.80it/s]
Loading 0:  38%|███▊      | 139/363 [00:51<01:13,  3.04it/s]
Loading 0:  39%|███▊      | 140/363 [00:51<01:09,  3.19it/s]
Loading 0:  39%|███▉      | 141/363 [00:51<01:00,  3.64it/s]
Loading 0:  39%|███▉      | 142/363 [00:51<00:52,  4.17it/s]
Loading 0:  39%|███▉      | 143/363 [00:52<01:27,  2.52it/s]
Loading 0:  40%|███▉      | 144/363 [00:53<01:57,  1.87it/s]
Loading 0:  40%|████      | 147/363 [00:54<01:27,  2.47it/s]
Loading 0:  41%|████      | 148/363 [00:55<01:46,  2.02it/s]
Loading 0:  41%|████      | 149/363 [00:56<02:06,  1.69it/s]
Loading 0:  42%|████▏     | 152/363 [00:56<01:14,  2.84it/s]
Loading 0:  42%|████▏     | 153/363 [00:56<01:09,  3.01it/s]
Loading 0:  42%|████▏     | 154/363 [00:57<01:00,  3.47it/s]
Loading 0:  43%|████▎     | 156/363 [00:57<01:10,  2.92it/s]
Loading 0:  43%|████▎     | 157/363 [00:58<01:33,  2.21it/s]
Loading 0:  44%|████▎     | 158/363 [00:59<01:55,  1.77it/s]
Loading 0:  44%|████▍     | 161/363 [00:59<01:07,  3.00it/s]
Loading 0:  45%|████▍     | 162/363 [01:00<01:03,  3.16it/s]
Loading 0:  45%|████▍     | 163/363 [01:00<00:54,  3.64it/s]
Loading 0:  45%|████▌     | 165/363 [01:01<01:06,  2.98it/s]
Loading 0:  46%|████▌     | 166/363 [01:02<01:28,  2.24it/s]
Loading 0:  46%|████▌     | 167/363 [01:02<01:49,  1.79it/s]
Loading 0:  47%|████▋     | 170/363 [01:03<01:03,  3.03it/s]
Loading 0:  47%|████▋     | 171/363 [01:03<01:00,  3.18it/s]
Loading 0:  47%|████▋     | 172/363 [01:03<00:52,  3.66it/s]
Loading 0:  48%|████▊     | 174/363 [01:04<01:03,  2.99it/s]
Loading 0:  48%|████▊     | 175/363 [01:05<01:23,  2.24it/s]
Loading 0:  48%|████▊     | 176/363 [01:06<01:44,  1.79it/s]
Loading 0:  49%|████▉     | 179/363 [01:06<01:00,  3.03it/s]
Loading 0:  50%|████▉     | 180/363 [01:06<00:57,  3.18it/s]
Loading 0:  50%|████▉     | 181/363 [01:06<00:49,  3.67it/s]
Loading 0:  50%|█████     | 183/363 [01:07<00:59,  3.01it/s]
Loading 0:  51%|█████     | 184/363 [01:08<01:19,  2.25it/s]
Loading 0:  51%|█████     | 185/363 [01:09<01:39,  1.80it/s]
Loading 0:  52%|█████▏    | 188/363 [01:09<00:57,  3.04it/s]
Loading 0:  52%|█████▏    | 189/363 [01:10<00:54,  3.20it/s]
Loading 0:  52%|█████▏    | 190/363 [01:10<00:47,  3.68it/s]
Loading 0:  53%|█████▎    | 192/363 [01:11<00:56,  3.01it/s]
Loading 0:  53%|█████▎    | 193/363 [01:12<01:15,  2.25it/s]
Loading 0:  53%|█████▎    | 194/363 [01:12<01:33,  1.80it/s]
Loading 0:  54%|█████▍    | 197/363 [01:13<00:54,  3.04it/s]
Loading 0:  55%|█████▍    | 198/363 [01:13<00:51,  3.20it/s]
Loading 0:  55%|█████▍    | 199/363 [01:13<00:44,  3.68it/s]
Loading 0:  55%|█████▌    | 201/363 [01:14<00:53,  3.01it/s]
Loading 0:  56%|█████▌    | 202/363 [01:15<01:11,  2.25it/s]
Loading 0:  56%|█████▌    | 203/363 [01:16<01:29,  1.79it/s]
Loading 0:  57%|█████▋    | 206/363 [01:16<00:51,  3.03it/s]
Loading 0:  57%|█████▋    | 207/363 [01:16<00:48,  3.19it/s]
Loading 0:  57%|█████▋    | 208/363 [01:16<00:42,  3.67it/s]
Loading 0:  58%|█████▊    | 210/363 [01:17<00:50,  3.00it/s]
Loading 0:  58%|█████▊    | 211/363 [01:18<01:07,  2.25it/s]
Loading 0:  58%|█████▊    | 212/363 [01:19<01:23,  1.80it/s]
Loading 0:  59%|█████▉    | 215/363 [01:19<00:48,  3.05it/s]
Loading 0:  60%|█████▉    | 216/363 [01:20<00:45,  3.20it/s]
Loading 0:  60%|█████▉    | 217/363 [01:20<00:39,  3.68it/s]
Loading 0:  60%|██████    | 218/363 [01:21<00:59,  2.44it/s]
Loading 0:  61%|██████    | 220/363 [01:21<00:43,  3.27it/s]
Loading 0:  61%|██████    | 221/363 [01:21<00:41,  3.42it/s]
Loading 0:  61%|██████    | 222/363 [01:21<00:35,  3.98it/s]
Loading 0:  61%|██████▏   | 223/363 [01:21<00:31,  4.49it/s]
Loading 0:  62%|██████▏   | 224/363 [01:22<00:54,  2.54it/s]
Loading 0:  62%|██████▏   | 225/363 [01:23<01:14,  1.86it/s]
Loading 0:  63%|██████▎   | 228/363 [01:24<00:54,  2.49it/s]
Loading 0:  63%|██████▎   | 229/363 [01:25<01:06,  2.02it/s]
Loading 0:  63%|██████▎   | 230/363 [01:26<01:19,  1.68it/s]
Loading 0:  64%|██████▍   | 233/363 [01:26<00:45,  2.85it/s]
Loading 0:  64%|██████▍   | 234/363 [01:26<00:42,  3.02it/s]
Loading 0:  65%|██████▍   | 235/363 [01:27<00:36,  3.51it/s]
Loading 0:  65%|██████▌   | 237/363 [01:27<00:42,  2.95it/s]
Loading 0:  66%|██████▌   | 238/363 [01:28<00:56,  2.23it/s]
Loading 0:  66%|██████▌   | 239/363 [01:29<01:09,  1.79it/s]
Loading 0:  67%|██████▋   | 242/363 [01:29<00:39,  3.03it/s]
Loading 0:  67%|██████▋   | 243/363 [01:30<00:37,  3.19it/s]
Loading 0:  67%|██████▋   | 244/363 [01:30<00:32,  3.67it/s]
Loading 0:  68%|██████▊   | 246/363 [01:31<00:38,  3.01it/s]
Loading 0:  68%|██████▊   | 247/363 [01:32<00:51,  2.26it/s]
Loading 0:  68%|██████▊   | 248/363 [01:32<01:03,  1.81it/s]
Loading 0:  69%|██████▉   | 251/363 [01:33<00:36,  3.06it/s]
Loading 0:  69%|██████▉   | 252/363 [01:33<00:34,  3.21it/s]
Loading 0:  70%|██████▉   | 253/363 [01:33<00:29,  3.70it/s]
Loading 0:  70%|███████   | 255/363 [01:34<00:35,  3.02it/s]
Loading 0:  71%|███████   | 256/363 [01:35<00:47,  2.26it/s]
Loading 0:  71%|███████   | 257/363 [01:36<00:58,  1.80it/s]
Loading 0:  72%|███████▏  | 260/363 [01:36<00:33,  3.04it/s]
Loading 0:  72%|███████▏  | 261/363 [01:36<00:31,  3.19it/s]
Loading 0:  72%|███████▏  | 262/363 [01:36<00:27,  3.68it/s]
Loading 0:  73%|███████▎  | 264/363 [01:37<00:32,  3.00it/s]
Loading 0:  73%|███████▎  | 265/363 [01:38<00:43,  2.24it/s]
Loading 0:  73%|███████▎  | 266/363 [01:39<00:54,  1.79it/s]
Loading 0:  74%|███████▍  | 269/363 [01:39<00:31,  3.03it/s]
Loading 0:  74%|███████▍  | 270/363 [01:40<00:29,  3.18it/s]
Loading 0:  75%|███████▍  | 271/363 [01:40<00:25,  3.66it/s]
Loading 0:  75%|███████▌  | 273/363 [01:41<00:30,  3.00it/s]
Loading 0:  75%|███████▌  | 274/363 [01:41<00:39,  2.24it/s]
Loading 0:  76%|███████▌  | 275/363 [01:42<00:49,  1.79it/s]
Loading 0:  77%|███████▋  | 278/363 [01:43<00:28,  3.02it/s]
Loading 0:  77%|███████▋  | 279/363 [01:43<00:26,  3.17it/s]
Loading 0:  77%|███████▋  | 280/363 [01:43<00:22,  3.65it/s]
Loading 0:  78%|███████▊  | 282/363 [01:44<00:27,  2.99it/s]
Loading 0:  78%|███████▊  | 283/363 [01:45<00:35,  2.24it/s]
Loading 0:  78%|███████▊  | 284/363 [01:46<00:44,  1.79it/s]
Loading 0:  79%|███████▉  | 287/363 [01:46<00:25,  3.02it/s]
Loading 0:  79%|███████▉  | 288/363 [01:46<00:23,  3.18it/s]
Loading 0:  80%|███████▉  | 289/363 [01:46<00:20,  3.66it/s]
Loading 0:  80%|████████  | 291/363 [01:47<00:24,  3.00it/s]
Loading 0:  80%|████████  | 292/363 [01:48<00:31,  2.24it/s]
Loading 0:  81%|████████  | 293/363 [01:49<00:39,  1.79it/s]
Loading 0:  82%|████████▏ | 296/363 [01:49<00:22,  3.02it/s]
Loading 0:  82%|████████▏ | 297/363 [01:50<00:20,  3.18it/s]
Loading 0:  82%|████████▏ | 298/363 [01:50<00:17,  3.66it/s]
Loading 0:  82%|████████▏ | 299/363 [01:51<00:26,  2.43it/s]
Loading 0:  83%|████████▎ | 301/363 [01:51<00:19,  3.25it/s]
Loading 0:  83%|████████▎ | 302/363 [01:51<00:17,  3.40it/s]
Loading 0:  83%|████████▎ | 303/363 [01:51<00:15,  3.91it/s]
Loading 0:  84%|████████▎ | 304/363 [01:52<00:15,  3.75it/s]
Loading 0:  84%|████████▍ | 306/363 [01:52<00:19,  2.97it/s]
Loading 0:  85%|████████▍ | 307/363 [01:53<00:26,  2.12it/s]
Loading 0:  85%|████████▌ | 310/363 [01:54<00:20,  2.63it/s]
Loading 0:  86%|████████▌ | 311/363 [01:55<00:24,  2.12it/s]
Loading 0:  86%|████████▌ | 312/363 [01:56<00:29,  1.76it/s]
Loading 0:  87%|████████▋ | 315/363 [01:56<00:16,  2.89it/s]
Loading 0:  87%|████████▋ | 316/363 [01:57<00:15,  3.05it/s]
Loading 0:  87%|████████▋ | 317/363 [01:57<00:13,  3.51it/s]
Loading 0:  88%|████████▊ | 319/363 [01:58<00:14,  2.95it/s]
Loading 0:  88%|████████▊ | 320/363 [01:58<00:19,  2.22it/s]
Loading 0:  88%|████████▊ | 321/363 [01:59<00:23,  1.79it/s]
Loading 0:  89%|████████▉ | 324/363 [02:00<00:12,  3.01it/s]
Loading 0:  90%|████████▉ | 325/363 [02:00<00:12,  3.16it/s]
Loading 0:  90%|████████▉ | 326/363 [02:00<00:10,  3.64it/s]
Loading 0:  90%|█████████ | 328/363 [02:01<00:11,  2.99it/s]
Loading 0:  91%|█████████ | 329/363 [02:02<00:15,  2.24it/s]
Loading 0:  91%|█████████ | 330/363 [02:03<00:18,  1.78it/s]
Loading 0:  92%|█████████▏| 333/363 [02:03<00:09,  3.02it/s]
Loading 0:  92%|█████████▏| 334/363 [02:03<00:09,  3.17it/s]
Loading 0:  92%|█████████▏| 335/363 [02:03<00:07,  3.65it/s]
Loading 0:  93%|█████████▎| 337/363 [02:04<00:08,  2.99it/s]
Loading 0:  93%|█████████▎| 338/363 [02:05<00:11,  2.24it/s]
Loading 0:  93%|█████████▎| 339/363 [02:06<00:13,  1.79it/s]
Loading 0:  94%|█████████▍| 342/363 [02:06<00:06,  3.02it/s]
Loading 0:  94%|█████████▍| 343/363 [02:07<00:06,  3.17it/s]
Loading 0:  95%|█████████▍| 344/363 [02:07<00:05,  3.66it/s]
Loading 0:  95%|█████████▌| 346/363 [02:08<00:05,  2.99it/s]
Loading 0:  96%|█████████▌| 347/363 [02:08<00:07,  2.24it/s]
Loading 0:  96%|█████████▌| 348/363 [02:09<00:08,  1.79it/s]
Loading 0:  97%|█████████▋| 351/363 [02:10<00:03,  3.03it/s]
Loading 0:  97%|█████████▋| 352/363 [02:10<00:03,  3.18it/s]
Loading 0:  97%|█████████▋| 353/363 [02:10<00:02,  3.66it/s]
Loading 0:  98%|█████████▊| 355/363 [02:11<00:02,  2.96it/s]
Loading 0:  98%|█████████▊| 356/363 [02:12<00:03,  2.22it/s]
Loading 0:  98%|█████████▊| 357/363 [02:13<00:03,  1.77it/s]
Loading 0:  99%|█████████▉| 360/363 [02:13<00:01,  2.99it/s]
Loading 0:  99%|█████████▉| 361/363 [02:13<00:00,  3.15it/s]
Loading 0: 100%|█████████▉| 362/363 [02:13<00:00,  3.63it/s]
                                                            
Job chaiml-grpo-nis-12bsft-56492-v15-mkmlizer completed after 235.3s with status: succeeded
Stopping job with name chaiml-grpo-nis-12bsft-56492-v15-mkmlizer
Pipeline stage MKMLizer completed in 236.07s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-grpo-nis-12bsft-56492-v15
Waiting for inference service chaiml-grpo-nis-12bsft-56492-v15 to be ready
Inference service chaiml-grpo-nis-12bsft-56492-v15 ready after 50.605429887771606s
Pipeline stage MKMLDeployer completed in 51.67s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.5847034454345703s
Received healthy response to inference request in 1.020550012588501s
Received healthy response to inference request in 1.3910017013549805s
Received healthy response to inference request in 1.4028584957122803s
5 requests
1 failed requests
5th percentile: 1.094640350341797
10th percentile: 1.1687306880950927
20th percentile: 1.3169113636016845
30th percentile: 1.3933730602264405
40th percentile: 1.3981157779693603
50th percentile: 1.4028584957122803
60th percentile: 1.4755964756011963
70th percentile: 1.5483344554901122
80th percentile: 5.3554633617401155
90th percentile: 12.896983194351197
95th percentile: 16.667743110656737
99th percentile: 19.68435104370117
mean time: 5.167523336410523
%s, retrying in %s seconds...
Received healthy response to inference request in 1.022322177886963s
Received healthy response to inference request in 1.0524990558624268s
Received healthy response to inference request in 1.5648694038391113s
Received healthy response to inference request in 1.196474552154541s
Received healthy response to inference request in 1.114454984664917s
5 requests
0 failed requests
5th percentile: 1.0283575534820557
10th percentile: 1.0343929290771485
20th percentile: 1.046463680267334
30th percentile: 1.064890241622925
40th percentile: 1.089672613143921
50th percentile: 1.114454984664917
60th percentile: 1.1472628116607666
70th percentile: 1.1800706386566162
80th percentile: 1.270153522491455
90th percentile: 1.4175114631652832
95th percentile: 1.4911904335021973
99th percentile: 1.5501336097717284
mean time: 1.1901240348815918
Pipeline stage StressChecker completed in 35.24s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.68s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.63s
Shutdown handler de-registered
chaiml-grpo-nis-12bsft_56492_v15 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-grpo-nis-12bsft-56492-v15-profiler
Waiting for inference service chaiml-grpo-nis-12bsft-56492-v15-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2439.29s
Shutdown handler de-registered
chaiml-grpo-nis-12bsft_56492_v15 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-nis-12bsft_56492_v15 status is now torndown due to DeploymentManager action