chaiml-cai-users-v0-f_42497

developer_uid: zonemercy
submission_id: chaiml-cai-users-v0-f_42497_v102
model_name: test
model_group: ChaiML/cai-users-v0-full
status: torndown
timestamp: 2025-11-23T02:51:19+00:00
num_battles: 5479
num_wins: 3181
celo_rating: 1345.0
family_friendly_score: 0.47240000000000004
family_friendly_standard_error: 0.007060286679731922
submission_type: basic
model_repo: ChaiML/cai-users-v0-full-sft-24b
model_architecture: MistralForCausalLM
model_num_parameters: 24096691200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 128
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.2775097496114251, 'latency_mean': 3.6033782947063444, 'latency_p50': 3.610092520713806, 'latency_p90': 3.7996519804000854}, {'batch_size': 2, 'throughput': 0.453193253420048, 'latency_mean': 4.41043234705925, 'latency_p50': 4.418305993080139, 'latency_p90': 4.626072263717651}, {'batch_size': 3, 'throughput': 0.5819300677574525, 'latency_mean': 5.13311008810997, 'latency_p50': 5.1555012464523315, 'latency_p90': 5.467145276069641}, {'batch_size': 5, 'throughput': 0.7342854654825722, 'latency_mean': 6.767156380414963, 'latency_p50': 6.756965756416321, 'latency_p90': 7.220879936218262}]
gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}
display_name: test
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/cai-users-v0-full-sft-24b
model_size: 24B
ranking_group: single
throughput_3p7s: 0.3
us_pacific_date: 2025-11-19
win_ratio: 0.5805803978828253
generation_params: {'temperature': 0.3, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', 'User:', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 128}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-cai-users-v0-f-42497-v102-mkmlizer
Waiting for job on chaiml-cai-users-v0-f-42497-v102-mkmlizer to finish
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  Version: 0.30.2                                                    ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  belonging to:                                                      ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-f-42497-v102-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-cai-users-v0-f-42497-v102-mkmlizer: Downloaded to shared memory in 61.093s
chaiml-cai-users-v0-f-42497-v102-mkmlizer: Checking if ChaiML/cai-users-v0-full-sft-24b already exists in ChaiML
chaiml-cai-users-v0-f-42497-v102-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpmn7n4oe_, device:0
chaiml-cai-users-v0-f-42497-v102-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-cai-users-v0-f-42497-v102-mkmlizer: quantized model in 309.848s
chaiml-cai-users-v0-f-42497-v102-mkmlizer: Processed model ChaiML/cai-users-v0-full-sft-24b in 370.942s
chaiml-cai-users-v0-f-42497-v102-mkmlizer: creating bucket guanaco-mkml-models
chaiml-cai-users-v0-f-42497-v102-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-cai-users-v0-f-42497-v102-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-cai-users-v0-f-42497-v102/nvidia
chaiml-cai-users-v0-f-42497-v102-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-cai-users-v0-f-42497-v102/nvidia/config.json
chaiml-cai-users-v0-f-42497-v102-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-cai-users-v0-f-42497-v102/nvidia/special_tokens_map.json
chaiml-cai-users-v0-f-42497-v102-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-cai-users-v0-f-42497-v102/nvidia/tokenizer_config.json
chaiml-cai-users-v0-f-42497-v102-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-cai-users-v0-f-42497-v102/nvidia/tokenizer.json
chaiml-cai-users-v0-f-42497-v102-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-cai-users-v0-f-42497-v102/nvidia/flywheel_model.1.safetensors
chaiml-cai-users-v0-f-42497-v102-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-cai-users-v0-f-42497-v102/nvidia/flywheel_model.0.safetensors
chaiml-cai-users-v0-f-42497-v102-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:02<04:17,  1.40it/s]
Loading 0:   1%|          | 4/363 [00:04<06:58,  1.17s/it]
Loading 0:   1%|▏         | 5/363 [00:06<08:54,  1.49s/it]
Loading 0:   2%|▏         | 8/363 [00:06<04:09,  1.43it/s]
Loading 0:   2%|▏         | 9/363 [00:07<03:38,  1.62it/s]
Loading 0:   3%|▎         | 10/363 [00:07<02:57,  1.99it/s]
Loading 0:   3%|▎         | 12/363 [00:09<04:14,  1.38it/s]
Loading 0:   4%|▎         | 13/363 [00:11<06:02,  1.04s/it]
Loading 0:   4%|▍         | 14/363 [00:13<07:39,  1.32s/it]
Loading 0:   5%|▍         | 17/363 [00:14<04:03,  1.42it/s]
Loading 0:   5%|▍         | 18/363 [00:14<03:34,  1.61it/s]
Loading 0:   5%|▌         | 19/363 [00:14<02:57,  1.94it/s]
Loading 0:   6%|▌         | 21/363 [00:16<04:07,  1.38it/s]
Loading 0:   6%|▌         | 22/363 [00:18<05:48,  1.02s/it]
Loading 0:   6%|▋         | 23/363 [00:20<07:21,  1.30s/it]
Loading 0:   7%|▋         | 26/363 [00:21<03:55,  1.43it/s]
Loading 0:   7%|▋         | 27/363 [00:21<03:27,  1.62it/s]
Loading 0:   8%|▊         | 28/363 [00:21<02:51,  1.96it/s]
Loading 0:   8%|▊         | 30/363 [00:21<02:06,  2.64it/s]
Loading 0:   9%|▊         | 31/363 [00:22<01:58,  2.81it/s]
Loading 0:   9%|▉         | 32/363 [00:22<01:41,  3.27it/s]
Loading 0:   9%|▉         | 33/363 [00:22<01:28,  3.73it/s]
Loading 0:   9%|▉         | 34/363 [00:24<04:08,  1.33it/s]
Loading 0:  10%|▉         | 35/363 [00:26<06:09,  1.13s/it]
Loading 0:  10%|▉         | 36/363 [00:28<07:47,  1.43s/it]
Loading 0:  11%|█         | 39/363 [00:31<05:35,  1.03s/it]
Loading 0:  11%|█         | 40/363 [00:33<06:46,  1.26s/it]
Loading 0:  11%|█▏        | 41/363 [00:35<07:54,  1.47s/it]
Loading 0:  12%|█▏        | 44/363 [00:35<04:15,  1.25it/s]
Loading 0:  12%|█▏        | 45/363 [00:35<03:42,  1.43it/s]
Loading 0:  13%|█▎        | 46/363 [00:36<03:02,  1.73it/s]
Loading 0:  13%|█▎        | 48/363 [00:38<03:59,  1.31it/s]
Loading 0:  13%|█▎        | 49/363 [00:40<05:29,  1.05s/it]
Loading 0:  14%|█▍        | 50/363 [00:42<06:52,  1.32s/it]
Loading 0:  15%|█▍        | 53/363 [00:42<03:41,  1.40it/s]
Loading 0:  15%|█▍        | 54/363 [00:43<03:14,  1.59it/s]
Loading 0:  15%|█▌        | 55/363 [00:43<02:40,  1.92it/s]
Loading 0:  16%|█▌        | 57/363 [00:45<03:42,  1.38it/s]
Loading 0:  16%|█▌        | 58/363 [00:47<05:12,  1.02s/it]
Loading 0:  16%|█▋        | 59/363 [00:49<06:34,  1.30s/it]
Loading 0:  17%|█▋        | 62/363 [00:49<03:30,  1.43it/s]
Loading 0:  17%|█▋        | 63/363 [00:50<03:04,  1.62it/s]
Loading 0:  18%|█▊        | 64/363 [00:50<02:33,  1.95it/s]
Loading 0:  18%|█▊        | 65/363 [00:52<04:24,  1.12it/s]
Loading 0:  18%|█▊        | 67/363 [00:52<02:56,  1.67it/s]
Loading 0:  19%|█▊        | 68/363 [00:53<02:34,  1.91it/s]
Loading 0:  19%|█▉        | 69/363 [00:53<02:05,  2.34it/s]
Loading 0:  19%|█▉        | 70/363 [00:53<01:43,  2.83it/s]
Loading 0:  20%|█▉        | 71/363 [00:55<04:01,  1.21it/s]
Loading 0:  20%|█▉        | 72/363 [00:57<05:44,  1.18s/it]
Loading 0:  20%|██        | 73/363 [00:59<07:06,  1.47s/it]
Loading 0:  21%|██        | 76/363 [01:00<03:27,  1.38it/s]
Loading 0:  21%|██        | 77/363 [01:00<03:00,  1.59it/s]
Loading 0:  21%|██▏       | 78/363 [01:00<02:27,  1.93it/s]
Loading 0:  22%|██▏       | 79/363 [01:02<04:18,  1.10it/s]
Loading 0:  22%|██▏       | 80/363 [01:04<05:51,  1.24s/it]
Loading 0:  23%|██▎       | 82/363 [01:05<03:40,  1.28it/s]
Loading 0:  23%|██▎       | 83/363 [01:05<03:05,  1.51it/s]
Loading 0:  23%|██▎       | 84/363 [01:05<02:27,  1.89it/s]
Loading 0:  24%|██▎       | 86/363 [01:07<03:28,  1.33it/s]
Loading 0:  24%|██▍       | 87/363 [01:09<04:59,  1.08s/it]
Loading 0:  25%|██▍       | 90/363 [01:11<04:04,  1.12it/s]
Loading 0:  25%|██▌       | 91/363 [01:14<05:07,  1.13s/it]
Loading 0:  25%|██▌       | 92/363 [01:16<06:07,  1.36s/it]
Loading 0:  26%|██▌       | 95/363 [01:16<03:22,  1.32it/s]
Loading 0:  26%|██▋       | 96/363 [01:16<02:57,  1.50it/s]
Loading 0:  27%|██▋       | 97/363 [01:17<02:28,  1.80it/s]
Loading 0:  27%|██▋       | 99/363 [01:19<03:17,  1.34it/s]
Loading 0:  28%|██▊       | 100/363 [01:21<04:32,  1.04s/it]
Loading 0:  28%|██▊       | 101/363 [01:23<05:41,  1.30s/it]
Loading 0:  29%|██▊       | 104/363 [01:23<03:03,  1.41it/s]
Loading 0:  29%|██▉       | 105/363 [01:24<02:40,  1.60it/s]
Loading 0:  29%|██▉       | 106/363 [01:24<02:13,  1.92it/s]
Loading 0:  29%|██▉       | 107/363 [01:24<01:50,  2.31it/s]
Loading 0:  30%|██▉       | 108/363 [01:26<03:38,  1.17it/s]
Loading 0:  31%|███       | 111/363 [01:28<03:16,  1.28it/s]
Loading 0:  31%|███       | 112/363 [01:30<04:22,  1.04s/it]
Loading 0:  31%|███       | 113/363 [01:32<05:24,  1.30s/it]
Loading 0:  32%|███▏      | 116/363 [01:33<02:58,  1.39it/s]
Loading 0:  32%|███▏      | 117/363 [01:33<02:36,  1.57it/s]
Loading 0:  33%|███▎      | 118/363 [01:33<02:09,  1.89it/s]
Loading 0:  33%|███▎      | 120/363 [01:35<02:57,  1.37it/s]
Loading 0:  33%|███▎      | 121/363 [01:37<04:08,  1.03s/it]
Loading 0:  34%|███▎      | 122/363 [01:40<05:13,  1.30s/it]
Loading 0:  34%|███▍      | 125/363 [01:40<02:48,  1.42it/s]
Loading 0:  35%|███▍      | 126/363 [01:40<02:27,  1.61it/s]
Loading 0:  35%|███▍      | 127/363 [01:40<02:02,  1.93it/s]
Loading 0:  36%|███▌      | 129/363 [01:42<02:49,  1.38it/s]
Loading 0:  36%|███▌      | 130/363 [01:45<03:58,  1.02s/it]
Loading 0:  36%|███▌      | 131/363 [01:47<05:01,  1.30s/it]
Loading 0:  37%|███▋      | 134/363 [01:47<02:41,  1.42it/s]
Loading 0:  37%|███▋      | 135/363 [01:47<02:21,  1.61it/s]
Loading 0:  37%|███▋      | 136/363 [01:48<01:57,  1.94it/s]
Loading 0:  38%|███▊      | 138/363 [01:50<02:43,  1.38it/s]
Loading 0:  38%|███▊      | 139/363 [01:52<03:49,  1.03s/it]
Loading 0:  39%|███▊      | 140/363 [01:54<04:50,  1.30s/it]
Loading 0:  39%|███▉      | 143/363 [01:54<02:35,  1.42it/s]
Loading 0:  40%|███▉      | 144/363 [01:55<02:16,  1.61it/s]
Loading 0:  40%|███▉      | 145/363 [01:55<01:52,  1.94it/s]
Loading 0:  40%|████      | 147/363 [01:55<01:22,  2.60it/s]
Loading 0:  41%|████      | 148/363 [01:55<01:17,  2.78it/s]
Loading 0:  41%|████      | 149/363 [01:55<01:05,  3.25it/s]
Loading 0:  41%|████▏     | 150/363 [01:56<00:57,  3.72it/s]
Loading 0:  42%|████▏     | 151/363 [01:58<02:40,  1.32it/s]
Loading 0:  42%|████▏     | 152/363 [02:00<03:58,  1.13s/it]
Loading 0:  42%|████▏     | 153/363 [02:02<05:01,  1.43s/it]
Loading 0:  43%|████▎     | 156/363 [02:04<03:34,  1.04s/it]
Loading 0:  43%|████▎     | 157/363 [02:06<04:19,  1.26s/it]
Loading 0:  44%|████▎     | 158/363 [02:09<05:02,  1.47s/it]
Loading 0:  44%|████▍     | 161/363 [02:09<02:41,  1.25it/s]
Loading 0:  45%|████▍     | 162/363 [02:09<02:20,  1.43it/s]
Loading 0:  45%|████▍     | 163/363 [02:09<01:55,  1.73it/s]
Loading 0:  45%|████▌     | 165/363 [02:11<02:31,  1.31it/s]
Loading 0:  46%|████▌     | 166/363 [02:13<03:27,  1.05s/it]
Loading 0:  46%|████▌     | 167/363 [02:16<04:19,  1.32s/it]
Loading 0:  47%|████▋     | 170/363 [02:16<02:18,  1.39it/s]
Loading 0:  47%|████▋     | 171/363 [02:16<02:01,  1.59it/s]
Loading 0:  47%|████▋     | 172/363 [02:16<01:40,  1.91it/s]
Loading 0:  48%|████▊     | 174/363 [02:19<02:18,  1.36it/s]
Loading 0:  48%|████▊     | 175/363 [02:21<03:15,  1.04s/it]
Loading 0:  48%|████▊     | 176/363 [02:23<04:05,  1.31s/it]
Loading 0:  49%|████▉     | 179/363 [02:23<02:10,  1.41it/s]
Loading 0:  50%|████▉     | 180/363 [02:24<01:54,  1.59it/s]
Loading 0:  50%|████▉     | 181/363 [02:24<01:34,  1.92it/s]
Loading 0:  50%|█████     | 182/363 [02:26<02:43,  1.11it/s]
Loading 0:  51%|█████     | 184/363 [02:26<01:48,  1.65it/s]
Loading 0:  51%|█████     | 185/363 [02:26<01:34,  1.88it/s]
Loading 0:  51%|█████     | 186/363 [02:27<01:17,  2.27it/s]
Loading 0:  52%|█████▏    | 187/363 [02:27<01:04,  2.73it/s]
Loading 0:  52%|█████▏    | 188/363 [02:29<02:27,  1.19it/s]
Loading 0:  52%|█████▏    | 189/363 [02:31<03:31,  1.22s/it]
Loading 0:  53%|█████▎    | 192/363 [02:33<02:41,  1.06it/s]
Loading 0:  53%|█████▎    | 193/363 [02:35<03:21,  1.19s/it]
Loading 0:  53%|█████▎    | 194/363 [02:38<03:59,  1.42s/it]
Loading 0:  54%|█████▍    | 197/363 [02:38<02:08,  1.29it/s]
Loading 0:  55%|█████▍    | 198/363 [02:38<01:52,  1.47it/s]
Loading 0:  55%|█████▍    | 199/363 [02:38<01:32,  1.78it/s]
Loading 0:  55%|█████▌    | 201/363 [02:40<02:02,  1.32it/s]
Loading 0:  56%|█████▌    | 202/363 [02:43<02:49,  1.05s/it]
Loading 0:  56%|█████▌    | 203/363 [02:45<03:31,  1.32s/it]
Loading 0:  57%|█████▋    | 206/363 [02:45<01:52,  1.40it/s]
Loading 0:  57%|█████▋    | 207/363 [02:45<01:38,  1.58it/s]
Loading 0:  57%|█████▋    | 208/363 [02:46<01:21,  1.91it/s]
Loading 0:  58%|█████▊    | 210/363 [02:48<01:52,  1.37it/s]
Loading 0:  58%|█████▊    | 211/363 [02:50<02:37,  1.03s/it]
Loading 0:  58%|█████▊    | 212/363 [02:52<03:17,  1.31s/it]
Loading 0:  59%|█████▉    | 215/363 [02:52<01:45,  1.41it/s]
Loading 0:  60%|█████▉    | 216/363 [02:53<01:31,  1.60it/s]
Loading 0:  60%|█████▉    | 217/363 [02:53<01:15,  1.93it/s]
Loading 0:  60%|██████    | 218/363 [02:55<02:10,  1.11it/s]
Loading 0:  60%|██████    | 219/363 [02:57<02:57,  1.23s/it]
Loading 0:  61%|██████    | 221/363 [02:57<01:51,  1.27it/s]
Loading 0:  61%|██████    | 222/363 [02:58<01:34,  1.50it/s]
Loading 0:  61%|██████▏   | 223/363 [02:58<01:15,  1.87it/s]
Loading 0:  62%|██████▏   | 224/363 [02:58<01:00,  2.30it/s]
Loading 0:  62%|██████▏   | 225/363 [03:00<02:03,  1.11it/s]
Loading 0:  63%|██████▎   | 228/363 [03:02<01:47,  1.25it/s]
Loading 0:  63%|██████▎   | 229/363 [03:04<02:23,  1.07s/it]
Loading 0:  63%|██████▎   | 230/363 [03:07<02:56,  1.33s/it]
Loading 0:  64%|██████▍   | 233/363 [03:07<01:34,  1.37it/s]
Loading 0:  64%|██████▍   | 234/363 [03:07<01:22,  1.56it/s]
Loading 0:  65%|██████▍   | 235/363 [03:07<01:08,  1.87it/s]
Loading 0:  65%|██████▌   | 237/363 [03:10<01:32,  1.36it/s]
Loading 0:  66%|██████▌   | 238/363 [03:12<02:09,  1.03s/it]
Loading 0:  66%|██████▌   | 239/363 [03:14<02:41,  1.30s/it]
Loading 0:  67%|██████▋   | 242/363 [03:14<01:25,  1.41it/s]
Loading 0:  67%|██████▋   | 243/363 [03:14<01:14,  1.60it/s]
Loading 0:  67%|██████▋   | 244/363 [03:15<01:02,  1.91it/s]
Loading 0:  68%|██████▊   | 246/363 [03:17<01:25,  1.37it/s]
Loading 0:  68%|██████▊   | 247/363 [03:19<01:59,  1.03s/it]
Loading 0:  68%|██████▊   | 248/363 [03:21<02:30,  1.31s/it]
Loading 0:  69%|██████▉   | 251/363 [03:21<01:19,  1.41it/s]
Loading 0:  69%|██████▉   | 252/363 [03:22<01:09,  1.60it/s]
Loading 0:  70%|██████▉   | 253/363 [03:22<00:56,  1.93it/s]
Loading 0:  70%|███████   | 255/363 [03:24<01:18,  1.38it/s]
Loading 0:  71%|███████   | 256/363 [03:26<01:49,  1.03s/it]
Loading 0:  71%|███████   | 257/363 [03:28<02:17,  1.30s/it]
Loading 0:  72%|███████▏  | 260/363 [03:29<01:12,  1.42it/s]
Loading 0:  72%|███████▏  | 261/363 [03:29<01:03,  1.61it/s]
Loading 0:  72%|███████▏  | 262/363 [03:29<00:52,  1.94it/s]
Loading 0:  73%|███████▎  | 264/363 [03:29<00:37,  2.61it/s]
Loading 0:  73%|███████▎  | 265/363 [03:30<00:35,  2.78it/s]
Loading 0:  73%|███████▎  | 266/363 [03:30<00:29,  3.28it/s]
Loading 0:  74%|███████▎  | 267/363 [03:30<00:25,  3.75it/s]
Loading 0:  74%|███████▍  | 268/363 [03:32<01:12,  1.32it/s]
Loading 0:  74%|███████▍  | 269/363 [03:34<01:46,  1.13s/it]
Loading 0:  74%|███████▍  | 270/363 [03:36<02:13,  1.44s/it]
Loading 0:  75%|███████▌  | 273/363 [03:53<05:29,  3.66s/it]
Loading 0:  75%|███████▌  | 274/363 [03:55<04:57,  3.35s/it]
Loading 0:  76%|███████▌  | 275/363 [03:57<04:31,  3.09s/it]
Loading 0:  77%|███████▋  | 278/363 [03:58<02:17,  1.61s/it]
Loading 0:  77%|███████▋  | 279/363 [03:58<01:53,  1.35s/it]
Loading 0:  77%|███████▋  | 280/363 [03:58<01:30,  1.09s/it]
Loading 0:  78%|███████▊  | 282/363 [04:00<01:26,  1.06s/it]
Loading 0:  78%|███████▊  | 283/363 [04:02<01:43,  1.29s/it]
Loading 0:  78%|███████▊  | 284/363 [04:04<01:58,  1.50s/it]
Loading 0:  79%|███████▉  | 287/363 [04:05<01:00,  1.25it/s]
Loading 0:  79%|███████▉  | 288/363 [04:05<00:51,  1.44it/s]
Loading 0:  80%|███████▉  | 289/363 [04:05<00:45,  1.64it/s]
Loading 0:  80%|████████  | 291/363 [04:07<00:54,  1.32it/s]
Loading 0:  80%|████████  | 292/363 [04:09<01:14,  1.05s/it]
Loading 0:  81%|████████  | 293/363 [04:11<01:31,  1.31s/it]
Loading 0:  82%|████████▏ | 296/363 [04:12<00:47,  1.42it/s]
Loading 0:  82%|████████▏ | 297/363 [04:12<00:40,  1.62it/s]
Loading 0:  82%|████████▏ | 298/363 [04:12<00:33,  1.95it/s]
Loading 0:  82%|████████▏ | 299/363 [04:14<00:55,  1.16it/s]
Loading 0:  83%|████████▎ | 301/363 [04:14<00:35,  1.74it/s]
Loading 0:  83%|████████▎ | 302/363 [04:15<00:30,  2.00it/s]
Loading 0:  83%|████████▎ | 303/363 [04:15<00:24,  2.44it/s]
Loading 0:  84%|████████▎ | 304/363 [04:15<00:20,  2.91it/s]
Loading 0:  84%|████████▍ | 305/363 [04:17<00:47,  1.22it/s]
Loading 0:  84%|████████▍ | 306/363 [04:19<01:08,  1.20s/it]
Loading 0:  85%|████████▌ | 309/363 [04:21<00:50,  1.07it/s]
Loading 0:  85%|████████▌ | 310/363 [04:24<01:02,  1.18s/it]
Loading 0:  86%|████████▌ | 311/363 [04:26<01:13,  1.41s/it]
Loading 0:  87%|████████▋ | 314/363 [04:26<00:37,  1.31it/s]
Loading 0:  87%|████████▋ | 315/363 [04:26<00:31,  1.51it/s]
Loading 0:  87%|████████▋ | 316/363 [04:26<00:25,  1.81it/s]
Loading 0:  88%|████████▊ | 318/363 [04:29<00:33,  1.34it/s]
Loading 0:  88%|████████▊ | 319/363 [04:31<00:45,  1.04s/it]
Loading 0:  88%|████████▊ | 320/363 [04:33<00:56,  1.31s/it]
Loading 0:  89%|████████▉ | 323/363 [04:33<00:28,  1.42it/s]
Loading 0:  89%|████████▉ | 324/363 [04:33<00:23,  1.63it/s]
Loading 0:  90%|████████▉ | 325/363 [04:34<00:19,  1.96it/s]
Loading 0:  90%|█████████ | 327/363 [04:36<00:26,  1.38it/s]
Loading 0:  90%|█████████ | 328/363 [04:38<00:35,  1.02s/it]
Loading 0:  91%|█████████ | 329/363 [04:40<00:44,  1.30s/it]
Loading 0:  91%|█████████▏| 332/363 [04:40<00:21,  1.43it/s]
Loading 0:  92%|█████████▏| 333/363 [04:41<00:18,  1.64it/s]
Loading 0:  92%|█████████▏| 334/363 [04:41<00:14,  1.98it/s]
Loading 0:  92%|█████████▏| 335/363 [04:43<00:24,  1.13it/s]
Loading 0:  93%|█████████▎| 336/363 [04:45<00:32,  1.22s/it]
Loading 0:  93%|█████████▎| 338/363 [04:45<00:19,  1.30it/s]
Loading 0:  93%|█████████▎| 339/363 [04:45<00:15,  1.54it/s]
Loading 0:  94%|█████████▎| 340/363 [04:46<00:11,  1.92it/s]
Loading 0:  94%|█████████▍| 341/363 [04:46<00:10,  2.13it/s]
Loading 0:  94%|█████████▍| 343/363 [04:48<00:14,  1.38it/s]
Loading 0:  95%|█████████▌| 346/363 [04:50<00:12,  1.39it/s]
Loading 0:  96%|█████████▌| 347/363 [04:52<00:15,  1.02it/s]
Loading 0:  96%|█████████▌| 348/363 [04:55<00:18,  1.24s/it]
Loading 0:  97%|█████████▋| 351/363 [04:55<00:08,  1.43it/s]
Loading 0:  97%|█████████▋| 352/363 [04:55<00:06,  1.62it/s]
Loading 0:  97%|█████████▋| 353/363 [04:55<00:05,  1.93it/s]
Loading 0:  98%|█████████▊| 355/363 [04:57<00:05,  1.37it/s]
Loading 0:  98%|█████████▊| 356/363 [05:00<00:07,  1.03s/it]
Loading 0:  98%|█████████▊| 357/363 [05:02<00:07,  1.30s/it]
Loading 0:  99%|█████████▉| 360/363 [05:02<00:02,  1.42it/s]
Loading 0:  99%|█████████▉| 361/363 [05:02<00:01,  1.62it/s]
Loading 0: 100%|█████████▉| 362/363 [05:02<00:00,  1.95it/s]
                                                            
Job chaiml-cai-users-v0-f-42497-v102-mkmlizer completed after 420.21s with status: succeeded
Stopping job with name chaiml-cai-users-v0-f-42497-v102-mkmlizer
Pipeline stage MKMLizer completed in 420.80s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-cai-users-v0-f-42497-v102
Waiting for inference service chaiml-cai-users-v0-f-42497-v102 to be ready
Inference service chaiml-cai-users-v0-f-42497-v102 ready after 151.43685221672058s
Pipeline stage MKMLDeployer completed in 152.21s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.5146706104278564s
Received healthy response to inference request in 3.1964292526245117s
Received healthy response to inference request in 3.6086699962615967s
Received healthy response to inference request in 3.204047918319702s
Received healthy response to inference request in 3.612241506576538s
5 requests
0 failed requests
5th percentile: 3.19795298576355
10th percentile: 3.199476718902588
20th percentile: 3.202524185180664
30th percentile: 3.266172456741333
40th percentile: 3.3904215335845946
50th percentile: 3.5146706104278564
60th percentile: 3.5522703647613527
70th percentile: 3.5898701190948485
80th percentile: 3.609384298324585
90th percentile: 3.6108129024505615
95th percentile: 3.61152720451355
99th percentile: 3.6120986461639406
mean time: 3.427211856842041
Pipeline stage StressChecker completed in 18.62s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.94s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.57s
Shutdown handler de-registered
chaiml-cai-users-v0-f_42497_v102 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3438.06s
Shutdown handler de-registered
chaiml-cai-users-v0-f_42497_v102 status is now inactive due to auto deactivation removed underperforming models
chaiml-cai-users-v0-f_42497_v102 status is now torndown due to DeploymentManager action