rirv938-devstral-cp624-_46237

developer_uid: rirv938

submission_id: rirv938-devstral-cp624-_46237_v1

model_name: rirv938-devstral-cp624-_46237_v1

model_group: rirv938/devstral_cp624_9

status: torndown

timestamp: 2025-07-08T15:10:54+00:00

num_battles: 10841

num_wins: 5725

celo_rating: 1316.46

family_friendly_score: 0.5349999999999999

family_friendly_standard_error: 0.007053722421530351

submission_type: basic

model_repo: rirv938/devstral_cp624_98ff_b35_r1_high_quality_merged

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.5164778462193698, 'latency_mean': 1.9359896874427795, 'latency_p50': 1.9280985593795776, 'latency_p90': 2.148467993736267}, {'batch_size': 3, 'throughput': 1.0384046447835324, 'latency_mean': 2.8786077797412872, 'latency_p50': 2.8548678159713745, 'latency_p90': 3.161208438873291}, {'batch_size': 5, 'throughput': 1.3082573404706024, 'latency_mean': 3.796555243730545, 'latency_p50': 3.827662706375122, 'latency_p90': 4.221749424934387}, {'batch_size': 6, 'throughput': 1.4132472572915127, 'latency_mean': 4.2240321779251095, 'latency_p50': 4.239426851272583, 'latency_p90': 4.711060976982116}, {'batch_size': 8, 'throughput': 1.5390429718290768, 'latency_mean': 5.14968310713768, 'latency_p50': 5.175149083137512, 'latency_p90': 5.71324405670166}, {'batch_size': 10, 'throughput': 1.6205061307602664, 'latency_mean': 6.11254253745079, 'latency_p50': 6.077062129974365, 'latency_p90': 6.831667494773865}]

gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}

display_name: rirv938-devstral-cp624-_46237_v1

is_internal_developer: True

language_model: rirv938/devstral_cp624_98ff_b35_r1_high_quality_merged

model_size: 24B

ranking_group: single

throughput_3p7s: 1.29

us_pacific_date: 2025-07-08

win_ratio: 0.5280878147772345

generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.45, 'frequency_penalty': 0.45, 'stopping_words': ['\n', 'You:', '</s>', '###'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{message}<|im_end|>\n', 'user_template': '<|im_start|>user\nYou:{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-devstral-cp624-46237-v1-mkmlizer
Waiting for job on rirv938-devstral-cp624-46237-v1-mkmlizer to finish
rirv938-devstral-cp624-46237-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-devstral-cp624-46237-v1-mkmlizer: ║                                                                     ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║                                                                     ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  Version: 0.29.15                                                   ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║                                                                     ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║                                                                     ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ║                                                                     ║
rirv938-devstral-cp624-46237-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-devstral-cp624-46237-v1-mkmlizer: Downloaded to shared memory in 278.349s
rirv938-devstral-cp624-46237-v1-mkmlizer: Checking if rirv938/devstral_cp624_98ff_b35_r1_high_quality_merged already exists in ChaiML
rirv938-devstral-cp624-46237-v1-mkmlizer: Creating repo ChaiML/devstral_cp624_98ff_b35_r1_high_quality_merged and uploading /tmp/tmpj5iqi_yv to it
rirv938-devstral-cp624-46237-v1-mkmlizer: 
  0%|          | 0/22 [00:00<?, ?it/s]
  5%|▍         | 1/22 [00:09<03:23,  9.71s/it]
  9%|▉         | 2/22 [00:16<02:38,  7.93s/it]
 14%|█▎        | 3/22 [00:24<02:32,  8.02s/it]
 18%|█▊        | 4/22 [00:28<01:53,  6.32s/it]
 23%|██▎       | 5/22 [00:36<01:57,  6.90s/it]
 27%|██▋       | 6/22 [00:43<01:52,  7.01s/it]
 32%|███▏      | 7/22 [00:47<01:33,  6.23s/it]
 36%|███▋      | 8/22 [00:54<01:28,  6.34s/it]
 41%|████      | 9/22 [01:01<01:23,  6.44s/it]
 45%|████▌     | 10/22 [01:09<01:23,  6.97s/it]
 50%|█████     | 11/22 [01:13<01:07,  6.17s/it]
 55%|█████▍    | 12/22 [01:17<00:53,  5.38s/it]
 59%|█████▉    | 13/22 [01:22<00:48,  5.36s/it]
 64%|██████▎   | 14/22 [01:28<00:44,  5.62s/it]
 68%|██████▊   | 15/22 [01:34<00:39,  5.59s/it]
 73%|███████▎  | 16/22 [01:38<00:30,  5.01s/it]
 77%|███████▋  | 17/22 [01:44<00:26,  5.30s/it]
 82%|████████▏ | 18/22 [01:47<00:19,  4.87s/it]
 86%|████████▋ | 19/22 [01:51<00:13,  4.41s/it]
 91%|█████████ | 20/22 [01:54<00:08,  4.19s/it]
 95%|█████████▌| 21/22 [02:03<00:05,  5.43s/it]
100%|██████████| 22/22 [02:04<00:00,  4.21s/it]
100%|██████████| 22/22 [02:04<00:00,  5.66s/it]
rirv938-devstral-cp624-46237-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpj5iqi_yv, device:0
rirv938-devstral-cp624-46237-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-devstral-cp624-46237-v1-mkmlizer: quantized model in 56.573s
rirv938-devstral-cp624-46237-v1-mkmlizer: Processed model rirv938/devstral_cp624_98ff_b35_r1_high_quality_merged in 551.706s
rirv938-devstral-cp624-46237-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-devstral-cp624-46237-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-devstral-cp624-46237-v1/nvidia/tokenizer_config.json
rirv938-devstral-cp624-46237-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-devstral-cp624-46237-v1/nvidia/tokenizer.json
rirv938-devstral-cp624-46237-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-devstral-cp624-46237-v1/nvidia/flywheel_model.1.safetensors
rirv938-devstral-cp624-46237-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-devstral-cp624-46237-v1/nvidia/flywheel_model.0.safetensors
rirv938-devstral-cp624-46237-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:00<00:12, 29.34it/s]
Loading 0:   2%|▏         | 6/363 [00:00<00:25, 13.95it/s]
Loading 0:   3%|▎         | 11/363 [00:00<00:16, 21.39it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:26, 12.98it/s]
Loading 0:   4%|▍         | 16/363 [00:01<00:27, 12.71it/s]
Loading 0:   6%|▌         | 21/363 [00:01<00:20, 16.46it/s]
Loading 0:   6%|▋         | 23/363 [00:01<00:25, 13.13it/s]
Loading 0:   8%|▊         | 28/363 [00:01<00:17, 19.15it/s]
Loading 0:   9%|▉         | 32/363 [00:01<00:14, 22.81it/s]
Loading 0:  10%|▉         | 35/363 [00:02<00:21, 15.21it/s]
Loading 0:  10%|█         | 38/363 [00:02<00:20, 15.86it/s]
Loading 0:  11%|█▏        | 41/363 [00:02<00:25, 12.42it/s]
Loading 0:  13%|█▎        | 46/363 [00:02<00:18, 17.57it/s]
Loading 0:  14%|█▍        | 50/363 [00:02<00:14, 21.15it/s]
Loading 0:  15%|█▍        | 53/363 [00:03<00:20, 15.27it/s]
Loading 0:  15%|█▌        | 56/363 [00:03<00:19, 15.88it/s]
Loading 0:  16%|█▋        | 59/363 [00:03<00:24, 12.59it/s]
Loading 0:  18%|█▊        | 64/363 [00:03<00:16, 17.62it/s]
Loading 0:  19%|█▊        | 68/363 [00:04<00:13, 21.25it/s]
Loading 0:  20%|█▉        | 71/363 [00:04<00:19, 15.01it/s]
Loading 0:  20%|██        | 74/363 [00:04<00:18, 15.69it/s]
Loading 0:  21%|██        | 77/363 [00:04<00:22, 12.48it/s]
Loading 0:  23%|██▎       | 82/363 [00:05<00:15, 17.64it/s]
Loading 0:  24%|██▎       | 86/363 [00:05<00:13, 21.15it/s]
Loading 0:  25%|██▍       | 90/363 [00:05<00:20, 13.50it/s]
Loading 0:  26%|██▌       | 95/363 [00:05<00:14, 17.99it/s]
Loading 0:  27%|██▋       | 99/363 [00:05<00:12, 21.09it/s]
Loading 0:  28%|██▊       | 103/363 [00:06<00:15, 16.59it/s]
Loading 0:  29%|██▉       | 106/363 [00:06<00:17, 14.81it/s]
Loading 0:  30%|███       | 109/363 [00:06<00:18, 13.62it/s]
Loading 0:  31%|███       | 111/363 [00:06<00:17, 14.19it/s]
Loading 0:  31%|███       | 113/363 [00:07<00:20, 11.96it/s]
Loading 0:  33%|███▎      | 118/363 [00:07<00:13, 17.77it/s]
Loading 0:  34%|███▎      | 122/363 [00:07<00:11, 21.16it/s]
Loading 0:  34%|███▍      | 125/363 [00:07<00:15, 14.96it/s]
Loading 0:  35%|███▌      | 128/363 [00:07<00:14, 15.73it/s]
Loading 0:  36%|███▌      | 131/363 [00:08<00:18, 12.42it/s]
Loading 0:  37%|███▋      | 136/363 [00:08<00:13, 17.45it/s]
Loading 0:  39%|███▊      | 140/363 [00:08<00:10, 20.60it/s]
Loading 0:  39%|███▉      | 143/363 [00:08<00:14, 14.95it/s]
Loading 0:  40%|████      | 146/363 [00:09<00:13, 15.72it/s]
Loading 0:  41%|████      | 149/363 [00:09<00:17, 12.53it/s]
Loading 0:  42%|████▏     | 154/363 [00:09<00:11, 17.72it/s]
Loading 0:  44%|████▎     | 158/363 [00:09<00:09, 21.31it/s]
Loading 0:  45%|████▍     | 162/363 [00:10<00:14, 13.83it/s]
Loading 0:  45%|████▌     | 165/363 [00:10<00:12, 15.52it/s]
Loading 0:  46%|████▋     | 168/363 [00:10<00:13, 14.39it/s]
Loading 0:  47%|████▋     | 172/363 [00:10<00:10, 18.10it/s]
Loading 0:  48%|████▊     | 176/363 [00:10<00:08, 21.84it/s]
Loading 0:  49%|████▉     | 179/363 [00:11<00:11, 15.97it/s]
Loading 0:  50%|█████     | 182/363 [00:11<00:10, 16.48it/s]
Loading 0:  51%|█████     | 185/363 [00:11<00:13, 12.83it/s]
Loading 0:  52%|█████▏    | 190/363 [00:11<00:09, 17.87it/s]
Loading 0:  53%|█████▎    | 194/363 [00:11<00:08, 21.01it/s]
Loading 0:  54%|█████▍    | 197/363 [00:12<00:10, 15.33it/s]
Loading 0:  55%|█████▌    | 200/363 [00:12<00:10, 15.95it/s]
Loading 0:  56%|█████▌    | 203/363 [00:27<03:42,  1.39s/it]
Loading 0:  57%|█████▋    | 207/363 [00:27<02:23,  1.08it/s]
Loading 0:  58%|█████▊    | 210/363 [00:27<01:45,  1.46it/s]
Loading 0:  59%|█████▊    | 213/363 [00:27<01:18,  1.92it/s]
Loading 0:  60%|█████▉    | 216/363 [00:27<00:59,  2.46it/s]
Loading 0:  60%|██████    | 219/363 [00:28<00:43,  3.33it/s]
Loading 0:  61%|██████    | 222/363 [00:28<00:33,  4.18it/s]
Loading 0:  62%|██████▏   | 226/363 [00:28<00:22,  6.12it/s]
Loading 0:  63%|██████▎   | 230/363 [00:28<00:15,  8.53it/s]
Loading 0:  64%|██████▍   | 233/363 [00:28<00:15,  8.60it/s]
Loading 0:  65%|██████▌   | 236/363 [00:29<00:12, 10.15it/s]
Loading 0:  66%|██████▌   | 239/363 [00:29<00:12,  9.69it/s]
Loading 0:  67%|██████▋   | 244/363 [00:29<00:08, 14.09it/s]
Loading 0:  68%|██████▊   | 248/363 [00:29<00:06, 17.24it/s]
Loading 0:  69%|██████▉   | 251/363 [00:30<00:08, 13.82it/s]
Loading 0:  70%|██████▉   | 254/363 [00:30<00:07, 14.72it/s]
Loading 0:  71%|███████   | 257/363 [00:30<00:08, 12.07it/s]
Loading 0:  72%|███████▏  | 262/363 [00:30<00:05, 17.15it/s]
Loading 0:  73%|███████▎  | 266/363 [00:30<00:04, 20.69it/s]
Loading 0:  74%|███████▍  | 269/363 [00:31<00:06, 15.25it/s]
Loading 0:  75%|███████▍  | 272/363 [00:31<00:05, 15.98it/s]
Loading 0:  76%|███████▌  | 275/363 [00:31<00:06, 12.74it/s]
Loading 0:  77%|███████▋  | 280/363 [00:31<00:04, 17.94it/s]
Loading 0:  78%|███████▊  | 284/363 [00:31<00:03, 21.42it/s]
Loading 0:  79%|███████▉  | 288/363 [00:32<00:05, 14.18it/s]
Loading 0:  80%|████████  | 291/363 [00:32<00:04, 16.22it/s]
Loading 0:  81%|████████  | 294/363 [00:32<00:04, 14.76it/s]
Loading 0:  82%|████████▏ | 298/363 [00:32<00:03, 18.55it/s]
Loading 0:  83%|████████▎ | 302/363 [00:32<00:02, 22.05it/s]
Loading 0:  84%|████████▍ | 305/363 [00:33<00:03, 15.91it/s]
Loading 0:  85%|████████▍ | 308/363 [00:33<00:03, 16.46it/s]
Loading 0:  86%|████████▌ | 311/363 [00:33<00:04, 12.77it/s]
Loading 0:  87%|████████▋ | 316/363 [00:33<00:02, 18.07it/s]
Loading 0:  88%|████████▊ | 320/363 [00:33<00:01, 21.62it/s]
Loading 0:  89%|████████▉ | 324/363 [00:34<00:02, 13.75it/s]
Loading 0:  90%|█████████ | 327/363 [00:34<00:02, 15.72it/s]
Loading 0:  91%|█████████ | 330/363 [00:34<00:02, 14.40it/s]
Loading 0:  92%|█████████▏| 334/363 [00:34<00:01, 18.06it/s]
Loading 0:  93%|█████████▎| 338/363 [00:35<00:01, 21.76it/s]
Loading 0:  94%|█████████▍| 341/363 [00:35<00:01, 15.90it/s]
Loading 0:  95%|█████████▍| 344/363 [00:35<00:01, 16.43it/s]
Loading 0:  96%|█████████▌| 347/363 [00:35<00:01, 12.72it/s]
Loading 0:  97%|█████████▋| 352/363 [00:36<00:00, 17.80it/s]
Loading 0:  98%|█████████▊| 356/363 [00:36<00:00, 21.16it/s]
Loading 0:  99%|█████████▉| 359/363 [00:36<00:00, 11.47it/s]
Loading 0: 100%|█████████▉| 362/363 [00:37<00:00, 11.45it/s]
                                                            
Job rirv938-devstral-cp624-46237-v1-mkmlizer completed after 581.77s with status: succeeded
Stopping job with name rirv938-devstral-cp624-46237-v1-mkmlizer
Pipeline stage MKMLizer completed in 582.31s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-devstral-cp624-46237-v1
Waiting for inference service rirv938-devstral-cp624-46237-v1 to be ready
Inference service rirv938-devstral-cp624-46237-v1 ready after 191.2535102367401s
Pipeline stage MKMLDeployer completed in 191.85s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.530776023864746s
Received healthy response to inference request in 2.4749157428741455s
Received healthy response to inference request in 2.725329637527466s
Received healthy response to inference request in 6.148494005203247s
Received healthy response to inference request in 9.903398513793945s
5 requests
0 failed requests
5th percentile: 2.4860877990722656
10th percentile: 2.4972598552703857
20th percentile: 2.519603967666626
30th percentile: 2.56968674659729
40th percentile: 2.6475081920623778
50th percentile: 2.725329637527466
60th percentile: 4.094595384597778
70th percentile: 5.46386113166809
80th percentile: 6.899474906921387
90th percentile: 8.401436710357666
95th percentile: 9.152417612075805
99th percentile: 9.753202333450318
mean time: 4.75658278465271
%s, retrying in %s seconds...
Received healthy response to inference request in 1.9841258525848389s
Received healthy response to inference request in 2.355896472930908s
Received healthy response to inference request in 1.9411113262176514s
Received healthy response to inference request in 6.001866579055786s
Received healthy response to inference request in 7.855457305908203s
5 requests
0 failed requests
5th percentile: 1.949714231491089
10th percentile: 1.9583171367645265
20th percentile: 1.9755229473114013
30th percentile: 2.058479976654053
40th percentile: 2.2071882247924806
50th percentile: 2.355896472930908
60th percentile: 3.814284515380859
70th percentile: 5.27267255783081
80th percentile: 6.37258472442627
90th percentile: 7.1140210151672365
95th percentile: 7.484739160537719
99th percentile: 7.781313676834106
mean time: 4.027691507339478
%s, retrying in %s seconds...
Received healthy response to inference request in 1.9196763038635254s
Received healthy response to inference request in 3.388749122619629s
Received healthy response to inference request in 19.408125162124634s
Received healthy response to inference request in 1.9593446254730225s
Received healthy response to inference request in 2.2612080574035645s
5 requests
0 failed requests
5th percentile: 1.9276099681854248
10th percentile: 1.9355436325073243
20th percentile: 1.951410961151123
30th percentile: 2.019717311859131
40th percentile: 2.1404626846313475
50th percentile: 2.2612080574035645
60th percentile: 2.7122244834899902
70th percentile: 3.163240909576416
80th percentile: 6.592624330520633
90th percentile: 13.000374746322633
95th percentile: 16.20424995422363
99th percentile: 18.76735012054443
mean time: 5.787420654296875
Pipeline stage StressChecker completed in 79.45s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.35s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.73s
Shutdown handler de-registered
rirv938-devstral-cp624-_46237_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3455.14s
Shutdown handler de-registered
rirv938-devstral-cp624-_46237_v1 status is now inactive due to auto deactivation removed underperforming models
rirv938-devstral-cp624-_46237_v1 status is now torndown due to DeploymentManager action