chaiml-mistral31-24b-s_69496

developer_uid: NischayDnk

submission_id: chaiml-mistral31-24b-s_69496_v33

model_name: chaiml-mistral31-24b-s_69496_v33

model_group: ChaiML/mistral31-24b-sim

status: torndown

timestamp: 2025-09-09T21:00:57+00:00

num_battles: 6060

num_wins: 3242

celo_rating: 1282.04

family_friendly_score: 0.4958

family_friendly_standard_error: 0.007070818340192315

submission_type: basic

model_repo: ChaiML/mistral31-24b-simpoexp1-s1-new-sft-retryv2top20lex-2e

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 10

max_input_tokens: 768

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.4520661583334164, 'latency_mean': 2.211970833539963, 'latency_p50': 2.232854127883911, 'latency_p90': 2.4603551626205444}, {'batch_size': 3, 'throughput': 0.9689764588730286, 'latency_mean': 3.0903012549877165, 'latency_p50': 3.103288769721985, 'latency_p90': 3.4460516214370727}, {'batch_size': 5, 'throughput': 1.2793811021249, 'latency_mean': 3.895700418949127, 'latency_p50': 3.936997652053833, 'latency_p90': 4.354672837257385}, {'batch_size': 6, 'throughput': 1.390145590359349, 'latency_mean': 4.283567234277725, 'latency_p50': 4.285313129425049, 'latency_p90': 4.828106451034546}, {'batch_size': 8, 'throughput': 1.5492463837534853, 'latency_mean': 5.119179023504257, 'latency_p50': 5.04189670085907, 'latency_p90': 5.86694221496582}, {'batch_size': 10, 'throughput': 1.6452188053511747, 'latency_mean': 6.025641187429428, 'latency_p50': 6.00773561000824, 'latency_p90': 6.9037573575973505}]

gpu_counts: {'NVIDIA L40S': 1}

display_name: chaiml-mistral31-24b-s_69496_v33

is_internal_developer: False

language_model: ChaiML/mistral31-24b-simpoexp1-s1-new-sft-retryv2top20lex-2e

model_size: 24B

ranking_group: single

throughput_3p7s: 1.22

us_pacific_date: 2025-09-09

win_ratio: 0.5349834983498349

generation_params: {'temperature': 0.45, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 60, 'presence_penalty': 0.4, 'frequency_penalty': 0.4, 'stopping_words': ['\n', '<|im_start|>', '<|im_end|>'], 'max_input_tokens': 768, 'best_of': 10, 'max_output_tokens': 64}

formatter: {'memory_template': '<|system|>Family Friendly{memory}\n', 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{message}<|im_end|>\n', 'user_template': '<|im_start|>user\nYou:{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-mistral31-24b-s-69496-v33-mkmlizer
Waiting for job on chaiml-mistral31-24b-s-69496-v33-mkmlizer to finish
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║                                                                     ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║                                                                     ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  Version: 0.30.2                                                    ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║                                                                     ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  belonging to:                                                      ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║                                                                     ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ║                                                                     ║
chaiml-mistral31-24b-s-69496-v33-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
HTTP Request: %s %s "%s %d %s"
chaiml-mistral31-24b-s-69496-v33-mkmlizer: Downloaded to shared memory in 30.608s
chaiml-mistral31-24b-s-69496-v33-mkmlizer: Checking if ChaiML/mistral31-24b-simpoexp1-s1-new-sft-retryv2top20lex-2e already exists in ChaiML
chaiml-mistral31-24b-s-69496-v33-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpms1ddu2s, device:0
chaiml-mistral31-24b-s-69496-v33-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-mistral31-24b-s-69496-v33-mkmlizer: quantized model in 42.086s
chaiml-mistral31-24b-s-69496-v33-mkmlizer: Processed model ChaiML/mistral31-24b-simpoexp1-s1-new-sft-retryv2top20lex-2e in 72.694s
chaiml-mistral31-24b-s-69496-v33-mkmlizer: creating bucket guanaco-mkml-models
chaiml-mistral31-24b-s-69496-v33-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-mistral31-24b-s-69496-v33-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v33/nvidia
chaiml-mistral31-24b-s-69496-v33-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v33/nvidia/config.json
chaiml-mistral31-24b-s-69496-v33-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v33/nvidia/special_tokens_map.json
chaiml-mistral31-24b-s-69496-v33-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v33/nvidia/tokenizer_config.json
chaiml-mistral31-24b-s-69496-v33-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v33/nvidia/tokenizer.json
chaiml-mistral31-24b-s-69496-v33-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v33/nvidia/flywheel_model.1.safetensors
chaiml-mistral31-24b-s-69496-v33-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v33/nvidia/flywheel_model.0.safetensors
chaiml-mistral31-24b-s-69496-v33-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:16, 21.75it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:08, 39.59it/s]
Loading 0:   5%|▍         | 17/363 [00:00<00:09, 37.03it/s]
Loading 0:   6%|▌         | 22/363 [00:00<00:09, 36.37it/s]
Loading 0:   7%|▋         | 26/363 [00:00<00:09, 37.25it/s]
Loading 0:   9%|▉         | 32/363 [00:00<00:07, 42.77it/s]
Loading 0:  10%|█         | 37/363 [00:01<00:12, 26.83it/s]
Loading 0:  11%|█▏        | 41/363 [00:01<00:13, 24.63it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:09, 32.10it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:10, 30.38it/s]
Loading 0:  16%|█▌        | 57/363 [00:01<00:09, 33.78it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:09, 31.49it/s]
Loading 0:  18%|█▊        | 65/363 [00:02<00:09, 32.72it/s]
Loading 0:  19%|█▉        | 70/363 [00:02<00:09, 29.43it/s]
Loading 0:  20%|██        | 74/363 [00:02<00:11, 26.26it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:09, 30.43it/s]
Loading 0:  23%|██▎       | 83/363 [00:02<00:08, 31.87it/s]
Loading 0:  24%|██▍       | 87/363 [00:02<00:09, 28.27it/s]
Loading 0:  25%|██▌       | 92/363 [00:03<00:09, 27.41it/s]
Loading 0:  27%|██▋       | 99/363 [00:03<00:07, 34.88it/s]
Loading 0:  28%|██▊       | 103/363 [00:03<00:08, 32.27it/s]
Loading 0:  29%|██▉       | 107/363 [00:03<00:09, 27.92it/s]
Loading 0:  31%|███       | 112/363 [00:03<00:08, 30.59it/s]
Loading 0:  32%|███▏      | 116/363 [00:03<00:07, 31.32it/s]
Loading 0:  33%|███▎      | 120/363 [00:03<00:07, 33.28it/s]
Loading 0:  34%|███▍      | 124/363 [00:03<00:07, 30.91it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:07, 33.41it/s]
Loading 0:  37%|███▋      | 133/363 [00:04<00:07, 31.27it/s]
Loading 0:  38%|███▊      | 138/363 [00:04<00:06, 34.70it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:07, 30.75it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:05, 37.49it/s]
Loading 0:  42%|████▏     | 153/363 [00:05<00:08, 24.42it/s]
Loading 0:  44%|████▎     | 158/363 [00:05<00:08, 24.68it/s]
Loading 0:  45%|████▌     | 165/363 [00:05<00:06, 31.53it/s]
Loading 0:  47%|████▋     | 169/363 [00:05<00:06, 30.37it/s]
Loading 0:  48%|████▊     | 174/363 [00:05<00:05, 32.81it/s]
Loading 0:  49%|████▉     | 178/363 [00:05<00:05, 31.03it/s]
Loading 0:  50%|█████     | 182/363 [00:05<00:05, 32.25it/s]
Loading 0:  52%|█████▏    | 187/363 [00:06<00:05, 29.43it/s]
Loading 0:  53%|█████▎    | 191/363 [00:06<00:05, 28.77it/s]
Loading 0:  54%|█████▎    | 195/363 [00:06<00:06, 25.98it/s]
Loading 0:  55%|█████▌    | 201/363 [00:19<02:12,  1.22it/s]
Loading 0:  56%|█████▌    | 203/363 [00:19<01:54,  1.40it/s]
Loading 0:  57%|█████▋    | 208/363 [00:19<01:13,  2.12it/s]
Loading 0:  58%|█████▊    | 211/363 [00:19<00:56,  2.67it/s]
Loading 0:  59%|█████▉    | 214/363 [00:19<00:43,  3.44it/s]
Loading 0:  60%|██████    | 218/363 [00:19<00:30,  4.82it/s]
Loading 0:  61%|██████    | 221/363 [00:20<00:23,  6.05it/s]
Loading 0:  62%|██████▏   | 224/363 [00:20<00:19,  7.25it/s]
Loading 0:  63%|██████▎   | 229/363 [00:20<00:12, 10.50it/s]
Loading 0:  64%|██████▍   | 232/363 [00:20<00:10, 12.36it/s]
Loading 0:  65%|██████▌   | 237/363 [00:20<00:07, 16.85it/s]
Loading 0:  66%|██████▋   | 241/363 [00:20<00:06, 18.95it/s]
Loading 0:  68%|██████▊   | 246/363 [00:20<00:04, 23.57it/s]
Loading 0:  69%|██████▉   | 250/363 [00:21<00:04, 24.62it/s]
Loading 0:  70%|███████   | 255/363 [00:21<00:03, 28.85it/s]
Loading 0:  71%|███████▏  | 259/363 [00:21<00:03, 28.22it/s]
Loading 0:  73%|███████▎  | 266/363 [00:21<00:02, 35.00it/s]
Loading 0:  74%|███████▍  | 270/363 [00:21<00:03, 23.43it/s]
Loading 0:  75%|███████▌  | 274/363 [00:21<00:03, 26.16it/s]
Loading 0:  77%|███████▋  | 278/363 [00:21<00:03, 27.14it/s]
Loading 0:  78%|███████▊  | 282/363 [00:22<00:02, 28.74it/s]
Loading 0:  79%|███████▉  | 286/363 [00:22<00:02, 27.50it/s]
Loading 0:  80%|████████  | 291/363 [00:22<00:02, 31.29it/s]
Loading 0:  81%|████████▏ | 295/363 [00:22<00:02, 29.81it/s]
Loading 0:  82%|████████▏ | 299/363 [00:22<00:02, 30.99it/s]
Loading 0:  84%|████████▎ | 304/363 [00:22<00:02, 28.59it/s]
Loading 0:  85%|████████▍ | 308/363 [00:22<00:01, 28.37it/s]
Loading 0:  86%|████████▌ | 311/363 [00:23<00:02, 23.68it/s]
Loading 0:  88%|████████▊ | 318/363 [00:23<00:01, 31.05it/s]
Loading 0:  89%|████████▊ | 322/363 [00:23<00:01, 29.49it/s]
Loading 0:  90%|█████████ | 327/363 [00:23<00:01, 32.71it/s]
Loading 0:  91%|█████████ | 331/363 [00:23<00:01, 30.75it/s]
Loading 0:  92%|█████████▏| 335/363 [00:23<00:00, 31.18it/s]
Loading 0:  93%|█████████▎| 339/363 [00:24<00:00, 30.72it/s]
Loading 0:  94%|█████████▍| 343/363 [00:24<00:01, 18.42it/s]
Loading 0:  96%|█████████▌| 348/363 [00:24<00:00, 20.05it/s]
Loading 0:  98%|█████████▊| 355/363 [00:24<00:00, 27.05it/s]
Loading 0:  99%|█████████▉| 359/363 [00:24<00:00, 26.40it/s]
                                                            
Job chaiml-mistral31-24b-s-69496-v33-mkmlizer completed after 114.39s with status: succeeded
Stopping job with name chaiml-mistral31-24b-s-69496-v33-mkmlizer
Pipeline stage MKMLizer completed in 114.84s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mistral31-24b-s-69496-v33
Waiting for inference service chaiml-mistral31-24b-s-69496-v33 to be ready
Failed to get response for submission mistralai-mistral-nem_93303_v569: ('http://mistralai-mistral-nem-93303-v569-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-mistral31-24b-s-69496-v33 ready after 90.23885774612427s
Pipeline stage MKMLDeployer completed in 90.64s
run pipeline stage %s
Running pipeline stage StressChecker
Failed to get response for submission mistralai-mistral-nem_93303_v569: ('http://mistralai-mistral-nem-93303-v569-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.5850627422332764s
Received healthy response to inference request in 2.6191277503967285s
Received healthy response to inference request in 2.213029384613037s
Received healthy response to inference request in 2.3952624797821045s
5 requests
1 failed requests
5th percentile: 2.2494760036468504
10th percentile: 2.285922622680664
20th percentile: 2.358815860748291
30th percentile: 2.433222532272339
40th percentile: 2.5091426372528076
50th percentile: 2.5850627422332764
60th percentile: 2.598688745498657
70th percentile: 2.612314748764038
80th percentile: 6.12011551856995
90th percentile: 13.122091054916384
95th percentile: 16.623078823089596
99th percentile: 19.423869037628172
mean time: 5.987309789657592
%s, retrying in %s seconds...
Received healthy response to inference request in 2.5171847343444824s
Received healthy response to inference request in 2.1574087142944336s
Received healthy response to inference request in 2.146380662918091s
Received healthy response to inference request in 2.078974962234497s
Received healthy response to inference request in 2.2466976642608643s
5 requests
0 failed requests
5th percentile: 2.092456102371216
10th percentile: 2.1059372425079346
20th percentile: 2.132899522781372
30th percentile: 2.1485862731933594
40th percentile: 2.1529974937438965
50th percentile: 2.1574087142944336
60th percentile: 2.193124294281006
70th percentile: 2.228839874267578
80th percentile: 2.300795078277588
90th percentile: 2.408989906311035
95th percentile: 2.463087320327759
99th percentile: 2.506365251541138
mean time: 2.2293293476104736
Pipeline stage StressChecker completed in 43.93s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.73s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.68s
Shutdown handler de-registered
chaiml-mistral31-24b-s_69496_v33 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-mistral31-24b-s-69496-v33-profiler
Waiting for inference service chaiml-mistral31-24b-s-69496-v33-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2922.25s
Shutdown handler de-registered
chaiml-mistral31-24b-s_69496_v33 status is now inactive due to auto deactivation removed underperforming models
chaiml-mistral31-24b-s_69496_v33 status is now torndown due to DeploymentManager action