chaiml-devstral-cp624-9_95911

developer_uid: rirv938

submission_id: chaiml-devstral-cp624-9_95911_v1

model_name: chaiml-devstral-cp624-9_95911_v1

model_group: ChaiML/devstral_cp624_98

status: torndown

timestamp: 2025-07-09T15:23:08+00:00

num_battles: 7902

num_wins: 4165

celo_rating: 1311.38

family_friendly_score: 0.536

family_friendly_standard_error: 0.007052715788970942

submission_type: basic

model_repo: ChaiML/devstral_cp624_98ff_b35_r1_high_quality_merged

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.5203650688915866, 'latency_mean': 1.9216220092773437, 'latency_p50': 1.90971040725708, 'latency_p90': 2.15964515209198}, {'batch_size': 3, 'throughput': 1.0532205221386168, 'latency_mean': 2.842905470132828, 'latency_p50': 2.8441364765167236, 'latency_p90': 3.1535149574279786}, {'batch_size': 5, 'throughput': 1.351647937564553, 'latency_mean': 3.68615518450737, 'latency_p50': 3.6824262142181396, 'latency_p90': 4.1571684837341305}, {'batch_size': 6, 'throughput': 1.4451413241125162, 'latency_mean': 4.1206897616386415, 'latency_p50': 4.1128997802734375, 'latency_p90': 4.665052723884583}, {'batch_size': 8, 'throughput': 1.5770133774751778, 'latency_mean': 5.0067680549621585, 'latency_p50': 5.014870047569275, 'latency_p90': 5.696548199653625}, {'batch_size': 10, 'throughput': 1.6715542901240044, 'latency_mean': 5.9420519948005674, 'latency_p50': 5.91376531124115, 'latency_p90': 6.537100863456726}]

gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}

display_name: chaiml-devstral-cp624-9_95911_v1

is_internal_developer: True

language_model: ChaiML/devstral_cp624_98ff_b35_r1_high_quality_merged

model_size: 24B

ranking_group: single

throughput_3p7s: 1.36

us_pacific_date: 2025-07-09

win_ratio: 0.5270817514553278

generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.45, 'frequency_penalty': 0.45, 'stopping_words': ['</s>', '\n', '###', 'You:'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{message}<|im_end|>\n', 'user_template': '<|im_start|>user\nYou:{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-devstral-cp624-9-95911-v1-mkmlizer
Waiting for job on chaiml-devstral-cp624-9-95911-v1-mkmlizer to finish
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║                                                                     ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║                                                                     ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  Version: 0.29.15                                                   ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║                                                                     ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║                                                                     ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ║                                                                     ║
chaiml-devstral-cp624-9-95911-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-gy-exp135-dpo-ex_31259_v1: HTTPConnectionPool(host='chaiml-gy-exp135-dpo-ex-31259-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-gy-exp135-dpo-ex_31259_v1: HTTPConnectionPool(host='chaiml-gy-exp135-dpo-ex-31259-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-devstral-cp624-9-95911-v1-mkmlizer: Downloaded to shared memory in 161.941s
chaiml-devstral-cp624-9-95911-v1-mkmlizer: Checking if ChaiML/devstral_cp624_98ff_b35_r1_high_quality_merged already exists in ChaiML
chaiml-devstral-cp624-9-95911-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpl9yaxw8a, device:0
chaiml-devstral-cp624-9-95911-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-devstral-cp624-9-95911-v1-mkmlizer: quantized model in 57.053s
chaiml-devstral-cp624-9-95911-v1-mkmlizer: Processed model ChaiML/devstral_cp624_98ff_b35_r1_high_quality_merged in 218.994s
chaiml-devstral-cp624-9-95911-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-devstral-cp624-9-95911-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-devstral-cp624-9-95911-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-devstral-cp624-9-95911-v1/nvidia/config.json
chaiml-devstral-cp624-9-95911-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-devstral-cp624-9-95911-v1/nvidia/special_tokens_map.json
chaiml-devstral-cp624-9-95911-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-devstral-cp624-9-95911-v1/nvidia/tokenizer_config.json
chaiml-devstral-cp624-9-95911-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-devstral-cp624-9-95911-v1/nvidia/tokenizer.json
chaiml-devstral-cp624-9-95911-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-devstral-cp624-9-95911-v1/nvidia/flywheel_model.1.safetensors
chaiml-devstral-cp624-9-95911-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-devstral-cp624-9-95911-v1/nvidia/flywheel_model.0.safetensors
chaiml-devstral-cp624-9-95911-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:00<00:12, 29.64it/s]
Loading 0:   2%|▏         | 6/363 [00:00<00:25, 14.12it/s]
Loading 0:   3%|▎         | 11/363 [00:00<00:16, 21.99it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:25, 13.89it/s]
Loading 0:   4%|▍         | 16/363 [00:01<00:26, 13.29it/s]
Loading 0:   6%|▌         | 21/363 [00:01<00:20, 16.74it/s]
Loading 0:   6%|▋         | 23/363 [00:01<00:25, 13.40it/s]
Loading 0:   8%|▊         | 28/363 [00:01<00:17, 19.46it/s]
Loading 0:   9%|▉         | 32/363 [00:01<00:14, 23.31it/s]
Loading 0:  10%|▉         | 36/363 [00:02<00:23, 14.04it/s]
Loading 0:  11%|█         | 39/363 [00:02<00:20, 16.09it/s]
Loading 0:  12%|█▏        | 42/363 [00:02<00:22, 14.50it/s]
Loading 0:  13%|█▎        | 46/363 [00:02<00:17, 18.15it/s]
Loading 0:  14%|█▍        | 50/363 [00:02<00:14, 22.05it/s]
Loading 0:  15%|█▍        | 53/363 [00:03<00:20, 15.48it/s]
Loading 0:  15%|█▌        | 56/363 [00:03<00:19, 16.03it/s]
Loading 0:  16%|█▋        | 59/363 [00:03<00:24, 12.46it/s]
Loading 0:  18%|█▊        | 64/363 [00:03<00:17, 17.33it/s]
Loading 0:  19%|█▊        | 68/363 [00:03<00:14, 20.57it/s]
Loading 0:  20%|█▉        | 71/363 [00:04<00:19, 14.88it/s]
Loading 0:  20%|██        | 74/363 [00:04<00:18, 15.49it/s]
Loading 0:  21%|██        | 77/363 [00:04<00:23, 12.26it/s]
Loading 0:  23%|██▎       | 82/363 [00:05<00:16, 17.06it/s]
Loading 0:  24%|██▎       | 86/363 [00:05<00:13, 20.07it/s]
Loading 0:  25%|██▍       | 89/363 [00:05<00:18, 14.66it/s]
Loading 0:  25%|██▌       | 92/363 [00:05<00:18, 14.97it/s]
Loading 0:  26%|██▋       | 96/363 [00:05<00:14, 18.74it/s]
Loading 0:  27%|██▋       | 99/363 [00:05<00:12, 20.60it/s]
Loading 0:  28%|██▊       | 102/363 [00:06<00:13, 18.76it/s]
Loading 0:  29%|██▉       | 105/363 [00:06<00:18, 14.14it/s]
Loading 0:  29%|██▉       | 107/363 [00:06<00:19, 13.06it/s]
Loading 0:  30%|███       | 109/363 [00:06<00:19, 12.80it/s]
Loading 0:  31%|███       | 111/363 [00:06<00:18, 13.75it/s]
Loading 0:  31%|███       | 113/363 [00:07<00:22, 11.33it/s]
Loading 0:  33%|███▎      | 118/363 [00:07<00:13, 17.79it/s]
Loading 0:  34%|███▎      | 122/363 [00:07<00:11, 21.55it/s]
Loading 0:  34%|███▍      | 125/363 [00:07<00:15, 15.07it/s]
Loading 0:  35%|███▌      | 128/363 [00:07<00:14, 15.74it/s]
Loading 0:  36%|███▌      | 131/363 [00:08<00:18, 12.42it/s]
Loading 0:  37%|███▋      | 136/363 [00:08<00:12, 17.71it/s]
Loading 0:  39%|███▊      | 140/363 [00:08<00:10, 21.36it/s]
Loading 0:  39%|███▉      | 143/363 [00:08<00:14, 15.41it/s]
Loading 0:  40%|████      | 146/363 [00:09<00:13, 16.10it/s]
Loading 0:  41%|████      | 149/363 [00:09<00:16, 12.78it/s]
Loading 0:  42%|████▏     | 154/363 [00:09<00:11, 18.03it/s]
Loading 0:  44%|████▎     | 158/363 [00:09<00:09, 21.57it/s]
Loading 0:  45%|████▍     | 162/363 [00:10<00:14, 14.03it/s]
Loading 0:  45%|████▌     | 165/363 [00:10<00:12, 16.03it/s]
Loading 0:  46%|████▋     | 168/363 [00:10<00:13, 14.56it/s]
Loading 0:  47%|████▋     | 172/363 [00:10<00:10, 18.15it/s]
Loading 0:  48%|████▊     | 176/363 [00:10<00:08, 21.80it/s]
Loading 0:  49%|████▉     | 179/363 [00:11<00:11, 15.66it/s]
Loading 0:  50%|█████     | 182/363 [00:11<00:11, 16.06it/s]
Loading 0:  51%|█████     | 185/363 [00:11<00:14, 12.59it/s]
Loading 0:  52%|█████▏    | 190/363 [00:11<00:09, 17.74it/s]
Loading 0:  53%|█████▎    | 194/363 [00:11<00:07, 21.39it/s]
Loading 0:  54%|█████▍    | 197/363 [00:12<00:10, 15.37it/s]
Loading 0:  55%|█████▌    | 200/363 [00:12<00:10, 16.04it/s]
Loading 0:  55%|█████▌    | 200/363 [00:26<00:10, 16.04it/s]
Loading 0:  55%|█████▌    | 201/363 [00:26<04:32,  1.68s/it]
Loading 0:  56%|█████▌    | 203/363 [00:27<03:29,  1.31s/it]
Loading 0:  57%|█████▋    | 207/363 [00:27<02:02,  1.28it/s]
Loading 0:  58%|█████▊    | 210/363 [00:27<01:25,  1.79it/s]
Loading 0:  59%|█████▊    | 213/363 [00:27<01:02,  2.38it/s]
Loading 0:  59%|█████▉    | 215/363 [00:27<00:51,  2.88it/s]
Loading 0:  60%|█████▉    | 217/363 [00:28<00:41,  3.52it/s]
Loading 0:  60%|██████    | 219/363 [00:28<00:32,  4.44it/s]
Loading 0:  61%|██████    | 221/363 [00:28<00:28,  4.99it/s]
Loading 0:  62%|██████▏   | 226/363 [00:28<00:15,  8.81it/s]
Loading 0:  63%|██████▎   | 230/363 [00:28<00:11, 12.00it/s]
Loading 0:  64%|██████▍   | 233/363 [00:29<00:12, 10.74it/s]
Loading 0:  65%|██████▌   | 236/363 [00:29<00:10, 11.86it/s]
Loading 0:  66%|██████▌   | 238/363 [00:29<00:11, 11.19it/s]
Loading 0:  66%|██████▌   | 240/363 [00:29<00:11, 11.10it/s]
Loading 0:  67%|██████▋   | 244/363 [00:29<00:07, 15.59it/s]
Loading 0:  68%|██████▊   | 248/363 [00:29<00:05, 19.60it/s]
Loading 0:  69%|██████▉   | 251/363 [00:30<00:07, 14.32it/s]
Loading 0:  70%|██████▉   | 254/363 [00:30<00:07, 15.23it/s]
Loading 0:  71%|███████   | 257/363 [00:30<00:08, 12.19it/s]
Loading 0:  72%|███████▏  | 262/363 [00:30<00:05, 17.52it/s]
Loading 0:  73%|███████▎  | 266/363 [00:30<00:04, 20.90it/s]
Loading 0:  74%|███████▍  | 269/363 [00:31<00:06, 15.08it/s]
Loading 0:  75%|███████▍  | 272/363 [00:31<00:05, 15.71it/s]
Loading 0:  76%|███████▌  | 275/363 [00:31<00:07, 12.28it/s]
Loading 0:  77%|███████▋  | 280/363 [00:31<00:04, 17.46it/s]
Loading 0:  78%|███████▊  | 284/363 [00:32<00:03, 21.02it/s]
Loading 0:  79%|███████▉  | 288/363 [00:32<00:05, 13.22it/s]
Loading 0:  80%|████████  | 291/363 [00:32<00:04, 15.16it/s]
Loading 0:  81%|████████  | 294/363 [00:33<00:04, 13.96it/s]
Loading 0:  82%|████████▏ | 298/363 [00:33<00:03, 17.46it/s]
Loading 0:  83%|████████▎ | 302/363 [00:33<00:02, 20.71it/s]
Loading 0:  84%|████████▍ | 305/363 [00:33<00:03, 15.05it/s]
Loading 0:  85%|████████▍ | 308/363 [00:33<00:03, 15.54it/s]
Loading 0:  86%|████████▌ | 311/363 [00:34<00:04, 12.26it/s]
Loading 0:  87%|████████▋ | 316/363 [00:34<00:02, 17.23it/s]
Loading 0:  88%|████████▊ | 320/363 [00:34<00:02, 20.78it/s]
Loading 0:  89%|████████▉ | 323/363 [00:34<00:02, 14.95it/s]
Loading 0:  90%|████████▉ | 326/363 [00:34<00:02, 15.74it/s]
Loading 0:  91%|█████████ | 329/363 [00:35<00:02, 12.48it/s]
Loading 0:  92%|█████████▏| 334/363 [00:35<00:01, 17.54it/s]
Loading 0:  93%|█████████▎| 338/363 [00:35<00:01, 21.15it/s]
Loading 0:  94%|█████████▍| 341/363 [00:35<00:01, 15.55it/s]
Loading 0:  95%|█████████▍| 344/363 [00:35<00:01, 16.24it/s]
Loading 0:  96%|█████████▌| 347/363 [00:36<00:01, 12.64it/s]
Loading 0:  97%|█████████▋| 352/363 [00:36<00:00, 17.47it/s]
Loading 0:  98%|█████████▊| 356/363 [00:36<00:00, 20.70it/s]
Loading 0:  99%|█████████▉| 359/363 [00:37<00:00, 11.23it/s]
Loading 0: 100%|█████████▉| 362/363 [00:37<00:00, 11.21it/s]
                                                            
Job chaiml-devstral-cp624-9-95911-v1-mkmlizer completed after 248.77s with status: succeeded
Stopping job with name chaiml-devstral-cp624-9-95911-v1-mkmlizer
Pipeline stage MKMLizer completed in 249.30s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-devstral-cp624-9-95911-v1
Waiting for inference service chaiml-devstral-cp624-9-95911-v1 to be ready
Inference service chaiml-devstral-cp624-9-95911-v1 ready after 200.66896295547485s
Pipeline stage MKMLDeployer completed in 201.43s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.569821357727051s
Received healthy response to inference request in 2.0300533771514893s
Received healthy response to inference request in 2.4571712017059326s
Received healthy response to inference request in 1.9657127857208252s
5 requests
1 failed requests
5th percentile: 1.978580904006958
10th percentile: 1.9914490222930907
20th percentile: 2.0171852588653563
30th percentile: 2.115476942062378
40th percentile: 2.286324071884155
50th percentile: 2.4571712017059326
60th percentile: 2.50223126411438
70th percentile: 2.547291326522827
80th percentile: 6.0868343353271515
90th percentile: 13.120860290527347
95th percentile: 16.63787326812744
99th percentile: 19.451483650207518
mean time: 5.835528993606568
%s, retrying in %s seconds...
Received healthy response to inference request in 1.831239938735962s
Received healthy response to inference request in 1.848602294921875s
Received healthy response to inference request in 1.7799413204193115s
Received healthy response to inference request in 2.2694308757781982s
Received healthy response to inference request in 2.1260900497436523s
5 requests
0 failed requests
5th percentile: 1.7902010440826417
10th percentile: 1.8004607677459716
20th percentile: 1.8209802150726317
30th percentile: 1.8347124099731444
40th percentile: 1.8416573524475097
50th percentile: 1.848602294921875
60th percentile: 1.959597396850586
70th percentile: 2.070592498779297
80th percentile: 2.1547582149505615
90th percentile: 2.21209454536438
95th percentile: 2.240762710571289
99th percentile: 2.2636972427368165
mean time: 1.9710608959197997
Pipeline stage StressChecker completed in 41.68s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.98s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.67s
Shutdown handler de-registered
chaiml-devstral-cp624-9_95911_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-devstral-cp624-9-95911-v1-profiler
Waiting for inference service chaiml-devstral-cp624-9-95911-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3137.15s
Shutdown handler de-registered
chaiml-devstral-cp624-9_95911_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-devstral-cp624-9_95911_v1 status is now torndown due to DeploymentManager action