developer_uid: zonemercy
submission_id: chaiml-cai-users-v0-fu_42497_v71
model_name: test
model_group: ChaiML/cai-users-v0-full
status: torndown
timestamp: 2025-10-31T11:21:36+00:00
num_battles: 7703
num_wins: 4362
celo_rating: 1317.41
family_friendly_score: 0.5044
family_friendly_standard_error: 0.007070794014818986
submission_type: basic
model_repo: ChaiML/cai-users-v0-full-sft-24b
model_architecture: MistralForCausalLM
model_num_parameters: 24096691200.0
best_of: 1
max_input_tokens: 2048
max_output_tokens: 128
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.2812707772762348, 'latency_mean': 3.5551950109004973, 'latency_p50': 3.5594383478164673, 'latency_p90': 3.7481106996536253}, {'batch_size': 2, 'throughput': 0.4652053315945099, 'latency_mean': 4.292412123680115, 'latency_p50': 4.296361804008484, 'latency_p90': 4.516993427276612}, {'batch_size': 3, 'throughput': 0.6127481087395288, 'latency_mean': 4.8739284014701845, 'latency_p50': 4.871999025344849, 'latency_p90': 5.108788585662841}, {'batch_size': 4, 'throughput': 0.7250992562181712, 'latency_mean': 5.510559329986572, 'latency_p50': 5.470355987548828, 'latency_p90': 6.23828489780426}, {'batch_size': 5, 'throughput': 0.8173528599615137, 'latency_mean': 6.081503292322159, 'latency_p50': 6.057632207870483, 'latency_p90': 6.393321371078491}]
gpu_counts: {'NVIDIA L40S': 1}
display_name: test
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/cai-users-v0-full-sft-24b
model_size: 24B
ranking_group: single
throughput_3p7s: 0.32
us_pacific_date: 2025-10-28
win_ratio: 0.5662728806958328
generation_params: {'temperature': 0.3, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', 'You:', 'User:'], 'max_input_tokens': 2048, 'best_of': 1, 'max_output_tokens': 128}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-cai-users-v0-fu-42497-v71-mkmlizer
Waiting for job on chaiml-cai-users-v0-fu-42497-v71-mkmlizer to finish
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  Version: 0.30.2                                                    ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  belonging to:                                                      ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: Downloaded to shared memory in 46.164s
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: Checking if ChaiML/cai-users-v0-full-sft-24b already exists in ChaiML
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpfez1tq8g, device:0
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: quantized model in 282.577s
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: Processed model ChaiML/cai-users-v0-full-sft-24b in 328.742s
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: creating bucket guanaco-mkml-models
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-cai-users-v0-fu-42497-v71/nvidia
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-cai-users-v0-fu-42497-v71/nvidia/config.json
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-cai-users-v0-fu-42497-v71/nvidia/special_tokens_map.json
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-cai-users-v0-fu-42497-v71/nvidia/tokenizer_config.json
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-cai-users-v0-fu-42497-v71/nvidia/flywheel_model.1.safetensors
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-cai-users-v0-fu-42497-v71/nvidia/flywheel_model.0.safetensors
chaiml-cai-users-v0-fu-42497-v71-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:01<03:56,  1.52it/s]
Loading 0:   1%|          | 4/363 [00:03<06:23,  1.07s/it]
Loading 0:   1%|▏         | 5/363 [00:05<08:10,  1.37s/it]
Loading 0:   2%|▏         | 8/363 [00:06<03:48,  1.56it/s]
Loading 0:   2%|▏         | 9/363 [00:06<03:20,  1.77it/s]
Loading 0:   3%|▎         | 10/363 [00:06<02:42,  2.17it/s]
Loading 0:   3%|▎         | 12/363 [00:08<03:53,  1.50it/s]
Loading 0:   4%|▎         | 13/363 [00:10<05:31,  1.05it/s]
Loading 0:   4%|▍         | 14/363 [00:12<07:00,  1.20s/it]
Loading 0:   5%|▍         | 17/363 [00:12<03:42,  1.55it/s]
Loading 0:   5%|▍         | 18/363 [00:13<03:15,  1.77it/s]
Loading 0:   5%|▌         | 19/363 [00:13<02:41,  2.13it/s]
Loading 0:   6%|▌         | 21/363 [00:15<03:46,  1.51it/s]
Loading 0:   6%|▌         | 22/363 [00:17<05:19,  1.07it/s]
Loading 0:   6%|▋         | 23/363 [00:19<06:45,  1.19s/it]
Loading 0:   7%|▋         | 26/363 [00:19<03:37,  1.55it/s]
Loading 0:   7%|▋         | 27/363 [00:19<03:10,  1.76it/s]
Loading 0:   8%|▊         | 28/363 [00:19<02:37,  2.13it/s]
Loading 0:   8%|▊         | 30/363 [00:20<01:54,  2.91it/s]
Loading 0:   9%|▊         | 31/363 [00:20<01:47,  3.10it/s]
Loading 0:   9%|▉         | 32/363 [00:20<01:30,  3.65it/s]
Loading 0:   9%|▉         | 33/363 [00:20<01:20,  4.11it/s]
Loading 0:   9%|▉         | 34/363 [00:22<03:46,  1.45it/s]
Loading 0:  10%|▉         | 35/363 [00:24<05:39,  1.04s/it]
Loading 0:  10%|▉         | 36/363 [00:26<07:08,  1.31s/it]
Loading 0:  11%|█         | 39/363 [00:28<05:07,  1.06it/s]
Loading 0:  11%|█         | 40/363 [00:30<06:18,  1.17s/it]
Loading 0:  11%|█▏        | 41/363 [00:32<07:19,  1.37s/it]
Loading 0:  12%|█▏        | 44/363 [00:32<03:56,  1.35it/s]
Loading 0:  12%|█▏        | 45/363 [00:33<03:25,  1.55it/s]
Loading 0:  13%|█▎        | 46/363 [00:33<02:48,  1.88it/s]
Loading 0:  13%|█▎        | 48/363 [00:35<03:40,  1.43it/s]
Loading 0:  13%|█▎        | 49/363 [00:36<05:01,  1.04it/s]
Loading 0:  14%|█▍        | 50/363 [00:38<06:17,  1.21s/it]
Loading 0:  15%|█▍        | 53/363 [00:39<03:22,  1.53it/s]
Loading 0:  15%|█▍        | 54/363 [00:39<02:57,  1.74it/s]
Loading 0:  15%|█▌        | 55/363 [00:39<02:26,  2.10it/s]
Loading 0:  16%|█▌        | 57/363 [00:41<03:23,  1.51it/s]
Loading 0:  16%|█▌        | 58/363 [00:43<04:45,  1.07it/s]
Loading 0:  16%|█▋        | 59/363 [00:45<06:00,  1.19s/it]
Loading 0:  17%|█▋        | 62/363 [00:45<03:13,  1.56it/s]
Loading 0:  17%|█▋        | 63/363 [00:46<02:49,  1.77it/s]
Loading 0:  18%|█▊        | 64/363 [00:46<02:20,  2.14it/s]
Loading 0:  18%|█▊        | 65/363 [00:48<04:02,  1.23it/s]
Loading 0:  18%|█▊        | 67/363 [00:48<02:42,  1.83it/s]
Loading 0:  19%|█▊        | 68/363 [00:48<02:21,  2.08it/s]
Loading 0:  19%|█▉        | 69/363 [00:48<01:55,  2.55it/s]
Loading 0:  19%|█▉        | 70/363 [00:48<01:34,  3.09it/s]
Loading 0:  20%|█▉        | 71/363 [00:50<03:41,  1.32it/s]
Loading 0:  20%|█▉        | 72/363 [00:52<05:15,  1.09s/it]
Loading 0:  20%|██        | 73/363 [00:54<06:31,  1.35s/it]
Loading 0:  21%|██        | 76/363 [00:55<03:10,  1.50it/s]
Loading 0:  21%|██        | 77/363 [00:55<02:45,  1.73it/s]
Loading 0:  21%|██▏       | 78/363 [00:55<02:14,  2.11it/s]
Loading 0:  22%|██▏       | 79/363 [00:57<03:56,  1.20it/s]
Loading 0:  22%|██▏       | 80/363 [00:59<05:22,  1.14s/it]
Loading 0:  23%|██▎       | 82/363 [00:59<03:21,  1.39it/s]
Loading 0:  23%|██▎       | 83/363 [00:59<02:50,  1.65it/s]
Loading 0:  23%|██▎       | 84/363 [01:00<02:15,  2.05it/s]
Loading 0:  24%|██▎       | 86/363 [01:02<03:10,  1.45it/s]
Loading 0:  24%|██▍       | 87/363 [01:04<04:33,  1.01it/s]
Loading 0:  25%|██▍       | 90/363 [01:06<03:44,  1.22it/s]
Loading 0:  25%|██▌       | 91/363 [01:07<04:41,  1.03s/it]
Loading 0:  25%|██▌       | 92/363 [01:09<05:36,  1.24s/it]
Loading 0:  26%|██▌       | 95/363 [01:10<03:06,  1.44it/s]
Loading 0:  26%|██▋       | 96/363 [01:10<02:43,  1.63it/s]
Loading 0:  27%|██▋       | 97/363 [01:10<02:15,  1.96it/s]
Loading 0:  27%|██▋       | 99/363 [01:12<03:00,  1.46it/s]
Loading 0:  28%|██▊       | 100/363 [01:14<04:09,  1.06it/s]
Loading 0:  28%|██▊       | 101/363 [01:16<05:12,  1.19s/it]
Loading 0:  29%|██▊       | 104/363 [01:16<02:48,  1.54it/s]
Loading 0:  29%|██▉       | 105/363 [01:17<02:27,  1.75it/s]
Loading 0:  29%|██▉       | 106/363 [01:17<02:01,  2.11it/s]
Loading 0:  29%|██▉       | 107/363 [01:17<01:41,  2.53it/s]
Loading 0:  30%|██▉       | 108/363 [01:19<03:20,  1.27it/s]
Loading 0:  31%|███       | 111/363 [01:21<03:00,  1.40it/s]
Loading 0:  31%|███       | 112/363 [01:23<04:00,  1.04it/s]
Loading 0:  31%|███       | 113/363 [01:25<04:58,  1.19s/it]
Loading 0:  32%|███▏      | 116/363 [01:25<02:43,  1.51it/s]
Loading 0:  32%|███▏      | 117/363 [01:25<02:23,  1.71it/s]
Loading 0:  33%|███▎      | 118/363 [01:25<01:59,  2.06it/s]
Loading 0:  33%|███▎      | 120/363 [01:27<02:43,  1.49it/s]
Loading 0:  33%|███▎      | 121/363 [01:29<03:47,  1.06it/s]
Loading 0:  34%|███▎      | 122/363 [01:31<04:47,  1.19s/it]
Loading 0:  34%|███▍      | 125/363 [01:32<02:34,  1.54it/s]
Loading 0:  35%|███▍      | 126/363 [01:32<02:15,  1.75it/s]
Loading 0:  35%|███▍      | 127/363 [01:32<01:51,  2.11it/s]
Loading 0:  36%|███▌      | 129/363 [01:34<02:35,  1.50it/s]
Loading 0:  36%|███▌      | 130/363 [01:36<03:38,  1.06it/s]
Loading 0:  36%|███▌      | 131/363 [01:38<04:36,  1.19s/it]
Loading 0:  37%|███▋      | 134/363 [01:38<02:27,  1.55it/s]
Loading 0:  37%|███▋      | 135/363 [01:38<02:09,  1.76it/s]
Loading 0:  37%|███▋      | 136/363 [01:39<01:47,  2.12it/s]
Loading 0:  38%|███▊      | 138/363 [01:40<02:29,  1.50it/s]
Loading 0:  38%|███▊      | 139/363 [01:42<03:30,  1.06it/s]
Loading 0:  39%|███▊      | 140/363 [01:44<04:25,  1.19s/it]
Loading 0:  39%|███▉      | 143/363 [01:45<02:22,  1.55it/s]
Loading 0:  40%|███▉      | 144/363 [01:45<02:04,  1.76it/s]
Loading 0:  40%|███▉      | 145/363 [01:45<01:43,  2.11it/s]
Loading 0:  40%|████      | 147/363 [01:45<01:16,  2.84it/s]
Loading 0:  41%|████      | 148/363 [01:46<01:10,  3.04it/s]
Loading 0:  41%|████      | 149/363 [01:46<01:00,  3.55it/s]
Loading 0:  41%|████▏     | 150/363 [01:46<00:52,  4.06it/s]
Loading 0:  42%|████▏     | 151/363 [01:48<02:27,  1.44it/s]
Loading 0:  42%|████▏     | 152/363 [01:50<03:39,  1.04s/it]
Loading 0:  42%|████▏     | 153/363 [01:52<04:36,  1.31s/it]
Loading 0:  43%|████▎     | 156/363 [01:54<03:16,  1.05it/s]
Loading 0:  43%|████▎     | 157/363 [01:56<03:58,  1.16s/it]
Loading 0:  44%|████▎     | 158/363 [01:58<04:36,  1.35s/it]
Loading 0:  44%|████▍     | 161/363 [01:58<02:28,  1.36it/s]
Loading 0:  45%|████▍     | 162/363 [01:58<02:08,  1.56it/s]
Loading 0:  45%|████▍     | 163/363 [01:58<01:46,  1.89it/s]
Loading 0:  45%|████▌     | 165/363 [02:00<02:18,  1.43it/s]
Loading 0:  46%|████▌     | 166/363 [02:02<03:10,  1.04it/s]
Loading 0:  46%|████▌     | 167/363 [02:04<03:57,  1.21s/it]
Loading 0:  47%|████▋     | 170/363 [02:05<02:06,  1.52it/s]
Loading 0:  47%|████▋     | 171/363 [02:05<01:50,  1.73it/s]
Loading 0:  47%|████▋     | 172/363 [02:05<01:31,  2.09it/s]
Loading 0:  48%|████▊     | 174/363 [02:07<02:06,  1.50it/s]
Loading 0:  48%|████▊     | 175/363 [02:09<02:56,  1.06it/s]
Loading 0:  48%|████▊     | 176/363 [02:11<03:42,  1.19s/it]
Loading 0:  49%|████▉     | 179/363 [02:11<01:58,  1.55it/s]
Loading 0:  50%|████▉     | 180/363 [02:12<01:44,  1.76it/s]
Loading 0:  50%|████▉     | 181/363 [02:12<01:25,  2.12it/s]
Loading 0:  50%|█████     | 182/363 [02:14<02:28,  1.22it/s]
Loading 0:  51%|█████     | 184/363 [02:14<01:38,  1.81it/s]
Loading 0:  51%|█████     | 185/363 [02:14<01:26,  2.06it/s]
Loading 0:  51%|█████     | 186/363 [02:14<01:10,  2.49it/s]
Loading 0:  52%|█████▏    | 187/363 [02:14<00:58,  3.02it/s]
Loading 0:  52%|█████▏    | 188/363 [02:16<02:13,  1.31it/s]
Loading 0:  52%|█████▏    | 189/363 [02:18<03:12,  1.11s/it]
Loading 0:  53%|█████▎    | 192/363 [02:20<02:26,  1.17it/s]
Loading 0:  53%|█████▎    | 193/363 [02:22<03:03,  1.08s/it]
Loading 0:  53%|█████▎    | 194/363 [02:24<03:38,  1.30s/it]
Loading 0:  54%|█████▍    | 197/363 [02:25<01:57,  1.41it/s]
Loading 0:  55%|█████▍    | 198/363 [02:25<01:42,  1.61it/s]
Loading 0:  55%|█████▍    | 199/363 [02:25<01:24,  1.95it/s]
Loading 0:  55%|█████▌    | 201/363 [02:27<01:51,  1.45it/s]
Loading 0:  56%|█████▌    | 202/363 [02:29<02:33,  1.05it/s]
Loading 0:  56%|█████▌    | 203/363 [02:31<03:12,  1.20s/it]
Loading 0:  57%|█████▋    | 206/363 [02:31<01:42,  1.53it/s]
Loading 0:  57%|█████▋    | 207/363 [02:31<01:29,  1.74it/s]
Loading 0:  57%|█████▋    | 208/363 [02:32<01:13,  2.10it/s]
Loading 0:  58%|█████▊    | 210/363 [02:33<01:41,  1.50it/s]
Loading 0:  58%|█████▊    | 211/363 [02:35<02:22,  1.07it/s]
Loading 0:  58%|█████▊    | 212/363 [02:37<02:59,  1.19s/it]
Loading 0:  59%|█████▉    | 215/363 [02:38<01:35,  1.55it/s]
Loading 0:  60%|█████▉    | 216/363 [02:38<01:23,  1.76it/s]
Loading 0:  60%|█████▉    | 217/363 [02:38<01:09,  2.11it/s]
Loading 0:  60%|██████    | 218/363 [02:40<01:58,  1.22it/s]
Loading 0:  60%|██████    | 219/363 [02:42<02:40,  1.12s/it]
Loading 0:  61%|██████    | 221/363 [02:42<01:41,  1.40it/s]
Loading 0:  61%|██████    | 222/363 [02:43<01:25,  1.65it/s]
Loading 0:  61%|██████▏   | 223/363 [02:43<01:08,  2.06it/s]
Loading 0:  62%|██████▏   | 224/363 [02:43<00:54,  2.54it/s]
Loading 0:  62%|██████▏   | 225/363 [02:45<01:52,  1.23it/s]
Loading 0:  63%|██████▎   | 228/363 [02:47<01:37,  1.38it/s]
Loading 0:  63%|██████▎   | 229/363 [02:49<02:10,  1.03it/s]
Loading 0:  63%|██████▎   | 230/363 [02:51<02:40,  1.21s/it]
Loading 0:  64%|██████▍   | 233/363 [02:51<01:25,  1.52it/s]
Loading 0:  64%|██████▍   | 234/363 [02:51<01:14,  1.72it/s]
Loading 0:  65%|██████▍   | 235/363 [02:51<01:01,  2.08it/s]
Loading 0:  65%|██████▌   | 237/363 [02:53<01:24,  1.50it/s]
Loading 0:  66%|██████▌   | 238/363 [02:55<01:57,  1.07it/s]
Loading 0:  66%|██████▌   | 239/363 [02:57<02:27,  1.19s/it]
Loading 0:  67%|██████▋   | 242/363 [02:57<01:17,  1.56it/s]
Loading 0:  67%|██████▋   | 243/363 [02:58<01:07,  1.77it/s]
Loading 0:  67%|██████▋   | 244/363 [02:58<00:55,  2.14it/s]
Loading 0:  68%|██████▊   | 246/363 [03:00<01:17,  1.51it/s]
Loading 0:  68%|██████▊   | 247/363 [03:02<01:48,  1.07it/s]
Loading 0:  68%|██████▊   | 248/363 [03:04<02:16,  1.19s/it]
Loading 0:  69%|██████▉   | 251/363 [03:04<01:12,  1.55it/s]
Loading 0:  69%|██████▉   | 252/363 [03:04<01:03,  1.76it/s]
Loading 0:  70%|██████▉   | 253/363 [03:04<00:51,  2.12it/s]
Loading 0:  70%|███████   | 255/363 [03:06<01:11,  1.51it/s]
Loading 0:  71%|███████   | 256/363 [03:08<01:40,  1.07it/s]
Loading 0:  71%|███████   | 257/363 [03:10<02:05,  1.19s/it]
Loading 0:  72%|███████▏  | 260/363 [03:11<01:06,  1.56it/s]
Loading 0:  72%|███████▏  | 261/363 [03:11<00:57,  1.77it/s]
Loading 0:  72%|███████▏  | 262/363 [03:11<00:47,  2.13it/s]
Loading 0:  73%|███████▎  | 264/363 [03:11<00:34,  2.86it/s]
Loading 0:  73%|███████▎  | 265/363 [03:12<00:32,  3.06it/s]
Loading 0:  73%|███████▎  | 266/363 [03:12<00:27,  3.55it/s]
Loading 0:  74%|███████▎  | 267/363 [03:12<00:23,  4.08it/s]
Loading 0:  74%|███████▍  | 268/363 [03:14<01:05,  1.45it/s]
Loading 0:  74%|███████▍  | 269/363 [03:16<01:36,  1.03s/it]
Loading 0:  74%|███████▍  | 270/363 [03:18<02:01,  1.31s/it]
Loading 0:  75%|███████▌  | 273/363 [03:32<04:55,  3.28s/it]
Loading 0:  75%|███████▌  | 274/363 [03:34<04:27,  3.00s/it]
Loading 0:  76%|███████▌  | 275/363 [03:36<04:05,  2.79s/it]
Loading 0:  77%|███████▋  | 278/363 [03:37<02:04,  1.46s/it]
Loading 0:  77%|███████▋  | 279/363 [03:37<01:42,  1.23s/it]
Loading 0:  77%|███████▋  | 280/363 [03:37<01:22,  1.01it/s]
Loading 0:  78%|███████▊  | 282/363 [03:39<01:18,  1.04it/s]
Loading 0:  78%|███████▊  | 283/363 [03:41<01:33,  1.17s/it]
Loading 0:  78%|███████▊  | 284/363 [03:43<01:47,  1.36s/it]
Loading 0:  79%|███████▉  | 287/363 [03:43<00:55,  1.37it/s]
Loading 0:  79%|███████▉  | 288/363 [03:43<00:47,  1.59it/s]
Loading 0:  80%|███████▉  | 289/363 [03:43<00:38,  1.93it/s]
Loading 0:  80%|████████  | 291/363 [03:45<00:47,  1.50it/s]
Loading 0:  80%|████████  | 292/363 [03:47<01:06,  1.07it/s]
Loading 0:  81%|████████  | 293/363 [03:49<01:23,  1.19s/it]
Loading 0:  82%|████████▏ | 296/363 [03:50<00:42,  1.56it/s]
Loading 0:  82%|████████▏ | 297/363 [03:50<00:36,  1.79it/s]
Loading 0:  82%|████████▏ | 298/363 [03:50<00:30,  2.16it/s]
Loading 0:  82%|████████▏ | 299/363 [03:52<00:49,  1.28it/s]
Loading 0:  83%|████████▎ | 301/363 [03:52<00:32,  1.92it/s]
Loading 0:  83%|████████▎ | 302/363 [03:52<00:27,  2.21it/s]
Loading 0:  83%|████████▎ | 303/363 [03:52<00:22,  2.68it/s]
Loading 0:  84%|████████▎ | 304/363 [03:52<00:18,  3.23it/s]
Loading 0:  84%|████████▍ | 305/363 [03:54<00:43,  1.34it/s]
Loading 0:  84%|████████▍ | 306/363 [03:56<01:02,  1.09s/it]
Loading 0:  85%|████████▌ | 309/363 [03:58<00:45,  1.17it/s]
Loading 0:  85%|████████▌ | 310/363 [04:00<00:56,  1.07s/it]
Loading 0:  86%|████████▌ | 311/363 [04:02<01:07,  1.29s/it]
Loading 0:  87%|████████▋ | 314/363 [04:03<00:34,  1.43it/s]
Loading 0:  87%|████████▋ | 315/363 [04:03<00:29,  1.65it/s]
Loading 0:  87%|████████▋ | 316/363 [04:03<00:23,  1.99it/s]
Loading 0:  88%|████████▊ | 318/363 [04:05<00:30,  1.47it/s]
Loading 0:  88%|████████▊ | 319/363 [04:07<00:41,  1.05it/s]
Loading 0:  88%|████████▊ | 320/363 [04:09<00:51,  1.19s/it]
Loading 0:  89%|████████▉ | 323/363 [04:09<00:25,  1.55it/s]
Loading 0:  89%|████████▉ | 324/363 [04:09<00:21,  1.78it/s]
Loading 0:  90%|████████▉ | 325/363 [04:09<00:17,  2.15it/s]
Loading 0:  90%|█████████ | 327/363 [04:11<00:23,  1.52it/s]
Loading 0:  90%|█████████ | 328/363 [04:13<00:32,  1.07it/s]
Loading 0:  91%|█████████ | 329/363 [04:15<00:40,  1.19s/it]
Loading 0:  91%|█████████▏| 332/363 [04:16<00:19,  1.56it/s]
Loading 0:  92%|█████████▏| 333/363 [04:16<00:16,  1.78it/s]
Loading 0:  92%|█████████▏| 334/363 [04:16<00:13,  2.15it/s]
Loading 0:  92%|█████████▏| 335/363 [04:18<00:22,  1.23it/s]
Loading 0:  93%|█████████▎| 336/363 [04:20<00:30,  1.11s/it]
Loading 0:  93%|█████████▎| 338/363 [04:20<00:17,  1.42it/s]
Loading 0:  93%|█████████▎| 339/363 [04:20<00:14,  1.68it/s]
Loading 0:  94%|█████████▎| 340/363 [04:20<00:10,  2.10it/s]
Loading 0:  94%|█████████▍| 341/363 [04:21<00:09,  2.33it/s]
Loading 0:  94%|█████████▍| 343/363 [04:23<00:13,  1.51it/s]
Loading 0:  95%|█████████▌| 346/363 [04:25<00:11,  1.52it/s]
Loading 0:  96%|█████████▌| 347/363 [04:27<00:14,  1.12it/s]
Loading 0:  96%|█████████▌| 348/363 [04:29<00:16,  1.13s/it]
Loading 0:  97%|█████████▋| 351/363 [04:29<00:07,  1.56it/s]
Loading 0:  97%|█████████▋| 352/363 [04:29<00:06,  1.76it/s]
Loading 0:  97%|█████████▋| 353/363 [04:29<00:04,  2.11it/s]
Loading 0:  98%|█████████▊| 355/363 [04:31<00:05,  1.50it/s]
Loading 0:  98%|█████████▊| 356/363 [04:33<00:06,  1.07it/s]
Loading 0:  98%|█████████▊| 357/363 [04:35<00:07,  1.18s/it]
Loading 0:  99%|█████████▉| 360/363 [04:35<00:01,  1.56it/s]
Loading 0:  99%|█████████▉| 361/363 [04:36<00:01,  1.76it/s]
Loading 0: 100%|█████████▉| 362/363 [04:36<00:00,  2.11it/s]
                                                            

Job chaiml-cai-users-v0-fu-42497-v71-mkmlizer completed after 415.44s with status: succeeded
Stopping job with name chaiml-cai-users-v0-fu-42497-v71-mkmlizer
Pipeline stage MKMLizer completed in 415.96s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-cai-users-v0-fu-42497-v71
Waiting for inference service chaiml-cai-users-v0-fu-42497-v71 to be ready
Inference service chaiml-cai-users-v0-fu-42497-v71 ready after 90.26358318328857s
Pipeline stage MKMLDeployer completed in 90.81s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.3322718143463135s
Received healthy response to inference request in 2.9692976474761963s
Received healthy response to inference request in 2.9705333709716797s
Received healthy response to inference request in 2.3209569454193115s
Received healthy response to inference request in 3.009526014328003s
5 requests
0 failed requests
5th percentile: 2.4506250858306884
10th percentile: 2.5802932262420653
20th percentile: 2.8396295070648194
30th percentile: 2.969544792175293
40th percentile: 2.9700390815734865
50th percentile: 2.9705333709716797
60th percentile: 2.986130428314209
70th percentile: 3.001727485656738
80th percentile: 3.074075174331665
90th percentile: 3.2031734943389893
95th percentile: 3.267722654342651
99th percentile: 3.319361982345581
mean time: 2.920517158508301
Pipeline stage StressChecker completed in 16.39s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.90s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.78s
Shutdown handler de-registered
chaiml-cai-users-v0-fu_42497_v71 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3027.98s
Shutdown handler de-registered
chaiml-cai-users-v0-fu_42497_v71 status is now inactive due to auto deactivation removed underperforming models
chaiml-cai-users-v0-fu_42497_v71 status is now torndown due to DeploymentManager action