chaiml-cai-users-v0-ful_84447

developer_uid: chai_backend_admin

submission_id: chaiml-cai-users-v0-ful_84447_v2

model_name: chaiml-cai-users-v0-ful_84447_v2

model_group: ChaiML/cai-users-v0-full

status: inactive

timestamp: 2025-10-11T05:50:52+00:00

num_battles: 6645

num_wins: 3557

celo_rating: 1306.3

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/cai-users-v0-full-sft-instruct-24b

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 1

max_input_tokens: 2048

max_output_tokens: 128

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.22685614398360696, 'latency_mean': 4.407953915596008, 'latency_p50': 4.412140130996704, 'latency_p90': 4.668945860862732}, {'batch_size': 2, 'throughput': 0.3832035708209122, 'latency_mean': 5.212160431146621, 'latency_p50': 5.228435754776001, 'latency_p90': 5.513266682624817}, {'batch_size': 5, 'throughput': 0.7013295259172934, 'latency_mean': 7.092540374994278, 'latency_p50': 7.0958662033081055, 'latency_p90': 7.76079375743866}]

gpu_counts: {'NVIDIA L40S': 1}

display_name: chaiml-cai-users-v0-ful_84447_v2

ineligible_reason: max_output_tokens!=64

is_internal_developer: True

language_model: ChaiML/cai-users-v0-full-sft-instruct-24b

model_size: 24B

ranking_group: single

us_pacific_date: 2025-10-10

win_ratio: 0.5352896914973665

generation_params: {'temperature': 0.85, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', 'User:', 'You:'], 'max_input_tokens': 2048, 'best_of': 1, 'max_output_tokens': 128}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': 'You: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': True}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-cai-users-v0-ful-84447-v2-mkmlizer
Waiting for job on chaiml-cai-users-v0-ful-84447-v2-mkmlizer to finish
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  Version: 0.30.2                                                    ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  belonging to:                                                      ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ║                                                                     ║
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: Downloaded to shared memory in 55.848s
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: Checking if ChaiML/cai-users-v0-full-sft-instruct-24b already exists in ChaiML
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpfwxs6eyj, device:0
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: quantized model in 42.312s
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: Processed model ChaiML/cai-users-v0-full-sft-instruct-24b in 98.160s
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: creating bucket guanaco-mkml-models
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-cai-users-v0-ful-84447-v2/nvidia
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-cai-users-v0-ful-84447-v2/nvidia/config.json
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-cai-users-v0-ful-84447-v2/nvidia/special_tokens_map.json
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-cai-users-v0-ful-84447-v2/nvidia/tokenizer.json
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-cai-users-v0-ful-84447-v2/nvidia/flywheel_model.1.safetensors
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-cai-users-v0-ful-84447-v2/nvidia/flywheel_model.0.safetensors
chaiml-cai-users-v0-ful-84447-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:16, 22.03it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:09, 37.75it/s]
Loading 0:   5%|▍         | 18/363 [00:00<00:08, 41.03it/s]
Loading 0:   6%|▋         | 23/363 [00:00<00:10, 31.85it/s]
Loading 0:   9%|▉         | 32/363 [00:00<00:07, 44.30it/s]
Loading 0:  10%|█         | 38/363 [00:01<00:10, 29.94it/s]
Loading 0:  12%|█▏        | 42/363 [00:01<00:11, 27.23it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:09, 32.35it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:10, 30.83it/s]
Loading 0:  16%|█▌        | 57/363 [00:01<00:08, 34.18it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:09, 32.14it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:08, 33.29it/s]
Loading 0:  19%|█▉        | 70/363 [00:02<00:10, 29.02it/s]
Loading 0:  20%|██        | 74/363 [00:02<00:11, 26.11it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:09, 30.14it/s]
Loading 0:  23%|██▎       | 83/363 [00:02<00:08, 31.19it/s]
Loading 0:  24%|██▍       | 87/363 [00:02<00:09, 27.64it/s]
Loading 0:  25%|██▌       | 92/363 [00:02<00:09, 27.11it/s]
Loading 0:  27%|██▋       | 99/363 [00:03<00:07, 34.50it/s]
Loading 0:  28%|██▊       | 103/363 [00:03<00:08, 32.33it/s]
Loading 0:  29%|██▉       | 107/363 [00:03<00:09, 27.92it/s]
Loading 0:  31%|███       | 112/363 [00:03<00:08, 30.33it/s]
Loading 0:  32%|███▏      | 116/363 [00:03<00:07, 30.95it/s]
Loading 0:  33%|███▎      | 120/363 [00:03<00:07, 32.17it/s]
Loading 0:  34%|███▍      | 124/363 [00:03<00:07, 30.72it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:06, 34.23it/s]
Loading 0:  37%|███▋      | 133/363 [00:04<00:07, 31.11it/s]
Loading 0:  38%|███▊      | 138/363 [00:04<00:06, 34.39it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:06, 32.23it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:05, 38.86it/s]
Loading 0:  42%|████▏     | 153/363 [00:04<00:08, 25.08it/s]
Loading 0:  44%|████▎     | 158/363 [00:05<00:08, 25.37it/s]
Loading 0:  45%|████▌     | 165/363 [00:05<00:06, 32.38it/s]
Loading 0:  47%|████▋     | 169/363 [00:05<00:06, 31.27it/s]
Loading 0:  48%|████▊     | 174/363 [00:05<00:05, 34.31it/s]
Loading 0:  49%|████▉     | 178/363 [00:05<00:05, 32.52it/s]
Loading 0:  50%|█████     | 182/363 [00:05<00:05, 33.16it/s]
Loading 0:  52%|█████▏    | 187/363 [00:05<00:05, 29.96it/s]
Loading 0:  53%|█████▎    | 191/363 [00:06<00:05, 29.19it/s]
Loading 0:  54%|█████▎    | 195/363 [00:06<00:06, 26.35it/s]
Loading 0:  55%|█████▌    | 201/363 [00:19<02:15,  1.20it/s]
Loading 0:  56%|█████▌    | 203/363 [00:19<01:56,  1.38it/s]
Loading 0:  57%|█████▋    | 208/363 [00:19<01:14,  2.08it/s]
Loading 0:  58%|█████▊    | 211/363 [00:19<00:57,  2.63it/s]
Loading 0:  59%|█████▉    | 215/363 [00:19<00:40,  3.67it/s]
Loading 0:  60%|██████    | 218/363 [00:20<00:30,  4.68it/s]
Loading 0:  61%|██████    | 221/363 [00:20<00:23,  5.96it/s]
Loading 0:  62%|██████▏   | 224/363 [00:20<00:19,  7.18it/s]
Loading 0:  63%|██████▎   | 229/363 [00:20<00:12, 10.52it/s]
Loading 0:  64%|██████▍   | 232/363 [00:20<00:10, 12.44it/s]
Loading 0:  65%|██████▌   | 237/363 [00:20<00:07, 16.93it/s]
Loading 0:  66%|██████▋   | 241/363 [00:20<00:06, 17.87it/s]
Loading 0:  67%|██████▋   | 244/363 [00:21<00:06, 19.49it/s]
Loading 0:  68%|██████▊   | 248/363 [00:21<00:06, 18.10it/s]
Loading 0:  70%|███████   | 255/363 [00:21<00:04, 25.34it/s]
Loading 0:  71%|███████▏  | 259/363 [00:21<00:04, 25.92it/s]
Loading 0:  73%|███████▎  | 266/363 [00:21<00:02, 32.85it/s]
Loading 0:  74%|███████▍  | 270/363 [00:22<00:04, 23.22it/s]
Loading 0:  76%|███████▌  | 275/363 [00:22<00:03, 23.95it/s]
Loading 0:  78%|███████▊  | 282/363 [00:22<00:02, 30.74it/s]
Loading 0:  79%|███████▉  | 286/363 [00:22<00:02, 29.90it/s]
Loading 0:  80%|████████  | 291/363 [00:22<00:02, 33.13it/s]
Loading 0:  81%|████████▏ | 295/363 [00:22<00:02, 31.56it/s]
Loading 0:  82%|████████▏ | 299/363 [00:22<00:01, 32.60it/s]
Loading 0:  84%|████████▎ | 304/363 [00:23<00:02, 28.42it/s]
Loading 0:  85%|████████▍ | 308/363 [00:23<00:01, 28.01it/s]
Loading 0:  86%|████████▌ | 311/363 [00:23<00:02, 24.03it/s]
Loading 0:  88%|████████▊ | 318/363 [00:23<00:01, 32.05it/s]
Loading 0:  89%|████████▊ | 322/363 [00:23<00:01, 30.79it/s]
Loading 0:  90%|█████████ | 327/363 [00:23<00:01, 33.99it/s]
Loading 0:  91%|█████████ | 331/363 [00:23<00:01, 31.93it/s]
Loading 0:  92%|█████████▏| 335/363 [00:24<00:00, 32.74it/s]
Loading 0:  93%|█████████▎| 339/363 [00:24<00:00, 32.46it/s]
Loading 0:  94%|█████████▍| 343/363 [00:24<00:01, 19.93it/s]
Loading 0:  96%|█████████▌| 348/363 [00:24<00:00, 21.54it/s]
Loading 0:  98%|█████████▊| 355/363 [00:24<00:00, 28.70it/s]
Loading 0:  99%|█████████▉| 359/363 [00:25<00:00, 28.42it/s]
                                                            
Job chaiml-cai-users-v0-ful-84447-v2-mkmlizer completed after 174.45s with status: succeeded
Stopping job with name chaiml-cai-users-v0-ful-84447-v2-mkmlizer
Pipeline stage MKMLizer completed in 174.91s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-cai-users-v0-ful-84447-v2
Waiting for inference service chaiml-cai-users-v0-ful-84447-v2 to be ready
Inference service chaiml-cai-users-v0-ful-84447-v2 ready after 70.56628274917603s
Pipeline stage MKMLDeployer completed in 71.04s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.959573268890381s
Received healthy response to inference request in 3.813678741455078s
Received healthy response to inference request in 3.811837911605835s
Received healthy response to inference request in 3.8151912689208984s
5 requests
1 failed requests
5th percentile: 3.8122060775756834
10th percentile: 3.8125742435455323
20th percentile: 3.8133105754852297
30th percentile: 3.8139812469482424
40th percentile: 3.8145862579345704
50th percentile: 3.8151912689208984
60th percentile: 3.8729440689086916
70th percentile: 3.9306968688964843
80th percentile: 7.195913743972781
90th percentile: 13.668594694137575
95th percentile: 16.904935169219968
99th percentile: 19.49400754928589
mean time: 7.108311367034912
%s, retrying in %s seconds...
Received healthy response to inference request in 3.3006556034088135s
Received healthy response to inference request in 3.7866005897521973s
Received healthy response to inference request in 2.8463003635406494s
Received healthy response to inference request in 1.4971060752868652s
Received healthy response to inference request in 3.7788355350494385s
5 requests
0 failed requests
5th percentile: 1.7669449329376221
10th percentile: 2.036783790588379
20th percentile: 2.5764615058898928
30th percentile: 2.9371714115142824
40th percentile: 3.118913507461548
50th percentile: 3.3006556034088135
60th percentile: 3.4919275760650637
70th percentile: 3.6831995487213134
80th percentile: 3.7803885459899904
90th percentile: 3.783494567871094
95th percentile: 3.7850475788116453
99th percentile: 3.786289987564087
mean time: 3.041899633407593
Pipeline stage StressChecker completed in 53.18s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.77s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.74s
Shutdown handler de-registered
chaiml-cai-users-v0-ful_84447_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3176.66s
Shutdown handler de-registered