developer_uid: zmeeks
submission_id: mistralai-mistral-nem_93303_v500
model_name: v1000
model_group: mistralai/Mistral-Nemo-I
status: torndown
timestamp: 2025-06-28T09:32:02+00:00
num_battles: 6695
num_wins: 3080
celo_rating: 1240.04
family_friendly_score: 0.628
family_friendly_standard_error: 0.006835437074540296
submission_type: basic
model_repo: mistralai/Mistral-Nemo-Instruct-2407
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.5915504114536789, 'latency_mean': 1.6902991425991059, 'latency_p50': 1.6773358583450317, 'latency_p90': 1.8612504720687866}, {'batch_size': 3, 'throughput': 1.059393878762689, 'latency_mean': 2.8184692645072937, 'latency_p50': 2.839889168739319, 'latency_p90': 3.0825665473937987}, {'batch_size': 5, 'throughput': 1.2642509986794945, 'latency_mean': 3.934078584909439, 'latency_p50': 3.9388636350631714, 'latency_p90': 4.3486947298049925}, {'batch_size': 6, 'throughput': 1.329814024546223, 'latency_mean': 4.489376883506775, 'latency_p50': 4.495574355125427, 'latency_p90': 5.043373322486877}, {'batch_size': 8, 'throughput': 1.3883355634739527, 'latency_mean': 5.7239905607700345, 'latency_p50': 5.652100205421448, 'latency_p90': 6.4876487255096436}, {'batch_size': 10, 'throughput': 1.3946606613176085, 'latency_mean': 7.096381485462189, 'latency_p50': 7.158616781234741, 'latency_p90': 7.886953496932984}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: v1000
is_internal_developer: False
language_model: mistralai/Mistral-Nemo-Instruct-2407
model_size: 13B
ranking_group: single
throughput_3p7s: 1.24
us_pacific_date: 2025-06-28
win_ratio: 0.46004480955937266
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '### Instruction:\n{memory}\n', 'prompt_template': '### Input:\n{prompt}\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nem-93303-v500-mkmlizer
Waiting for job on mistralai-mistral-nem-93303-v500-mkmlizer to finish
mistralai-mistral-nem-93303-v500-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-nem-93303-v500-mkmlizer: ║                                                                     ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║                                                                     ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  Version: 0.29.3                                                    ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║                                                                     ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  belonging to:                                                      ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║                                                                     ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
mistralai-mistral-nem-93303-v500-mkmlizer: ║                                                                     ║
mistralai-mistral-nem-93303-v500-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
mistralai-mistral-nem-93303-v500-mkmlizer: Downloaded to shared memory in 45.967s
mistralai-mistral-nem-93303-v500-mkmlizer: Checking if mistralai/Mistral-Nemo-Instruct-2407 already exists in ChaiML
mistralai-mistral-nem-93303-v500-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpyo2fnqig, device:0
mistralai-mistral-nem-93303-v500-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nem-93303-v500-mkmlizer: quantized model in 31.313s
mistralai-mistral-nem-93303-v500-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 77.365s
mistralai-mistral-nem-93303-v500-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nem-93303-v500-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nem-93303-v500-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v500
mistralai-mistral-nem-93303-v500-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v500/special_tokens_map.json
mistralai-mistral-nem-93303-v500-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v500/config.json
mistralai-mistral-nem-93303-v500-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v500/tokenizer_config.json
mistralai-mistral-nem-93303-v500-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v500/tokenizer.json
mistralai-mistral-nem-93303-v500-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 30.06it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:07, 48.01it/s]
Loading 0:   5%|▍         | 18/363 [00:00<00:07, 47.53it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 39.40it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 44.90it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:07, 43.74it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:07, 43.15it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:07, 41.00it/s]
Loading 0:  14%|█▍        | 51/363 [00:01<00:07, 40.00it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:07, 41.23it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:10, 30.00it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:09, 29.80it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:08, 34.63it/s]
Loading 0:  21%|██        | 75/363 [00:01<00:08, 34.28it/s]
Loading 0:  22%|██▏       | 80/363 [00:02<00:07, 37.41it/s]
Loading 0:  23%|██▎       | 84/363 [00:02<00:07, 36.42it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:07, 38.85it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 39.11it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:06, 40.07it/s]
Loading 0:  29%|██▊       | 104/363 [00:02<00:06, 41.59it/s]
Loading 0:  30%|███       | 109/363 [00:02<00:05, 42.58it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:06, 35.82it/s]
Loading 0:  33%|███▎      | 118/363 [00:03<00:07, 34.14it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:05, 41.31it/s]
Loading 0:  36%|███▌      | 130/363 [00:03<00:05, 41.09it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 40.84it/s]
Loading 0:  39%|███▊      | 140/363 [00:03<00:05, 40.87it/s]
Loading 0:  40%|███▉      | 145/363 [00:03<00:08, 25.35it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:08, 25.70it/s]
Loading 0:  43%|████▎     | 156/363 [00:04<00:06, 32.69it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 33.09it/s]
Loading 0:  45%|████▌     | 165/363 [00:04<00:05, 35.88it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 35.35it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:04, 38.23it/s]
Loading 0:  49%|████▉     | 179/363 [00:04<00:04, 38.69it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:04, 38.74it/s]
Loading 0:  52%|█████▏    | 188/363 [00:05<00:04, 38.39it/s]
Loading 0:  53%|█████▎    | 192/363 [00:05<00:04, 38.57it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:04, 37.77it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:04, 40.36it/s]
Loading 0:  57%|█████▋    | 206/363 [00:05<00:03, 39.97it/s]
Loading 0:  58%|█████▊    | 211/363 [00:05<00:03, 40.59it/s]
Loading 0:  60%|█████▉    | 216/363 [00:05<00:03, 42.40it/s]
Loading 0:  61%|██████    | 221/363 [00:05<00:03, 43.66it/s]
Loading 0:  62%|██████▏   | 226/363 [00:06<00:05, 25.94it/s]
Loading 0:  63%|██████▎   | 230/363 [00:06<00:05, 26.50it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 33.50it/s]
Loading 0:  66%|██████▋   | 241/363 [00:06<00:03, 33.76it/s]
Loading 0:  68%|██████▊   | 246/363 [00:06<00:03, 36.66it/s]
Loading 0:  69%|██████▉   | 251/363 [00:06<00:03, 37.08it/s]
Loading 0:  70%|███████   | 255/363 [00:06<00:02, 37.35it/s]
Loading 0:  71%|███████▏  | 259/363 [00:07<00:02, 36.77it/s]
Loading 0:  73%|███████▎  | 264/363 [00:07<00:02, 39.61it/s]
Loading 0:  74%|███████▍  | 269/363 [00:07<00:02, 39.20it/s]
Loading 0:  75%|███████▌  | 274/363 [00:07<00:02, 37.63it/s]
Loading 0:  77%|███████▋  | 278/363 [00:07<00:02, 37.38it/s]
Loading 0:  78%|███████▊  | 283/363 [00:07<00:02, 38.51it/s]
Loading 0:  79%|███████▉  | 287/363 [00:07<00:02, 37.87it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 37.22it/s]
Loading 0:  81%|████████▏ | 295/363 [00:08<00:01, 35.65it/s]
Loading 0:  82%|████████▏ | 299/363 [00:08<00:01, 35.15it/s]
Loading 0:  84%|████████▎ | 304/363 [00:08<00:03, 18.87it/s]
Loading 0:  85%|████████▍ | 307/363 [00:08<00:02, 20.19it/s]
Loading 0:  86%|████████▌ | 312/363 [00:08<00:02, 22.99it/s]
Loading 0:  88%|████████▊ | 319/363 [00:09<00:01, 31.68it/s]
Loading 0:  89%|████████▉ | 324/363 [00:09<00:01, 34.40it/s]
Loading 0:  91%|█████████ | 329/363 [00:09<00:00, 36.77it/s]
Loading 0:  92%|█████████▏| 334/363 [00:09<00:00, 38.88it/s]
Loading 0:  93%|█████████▎| 339/363 [00:09<00:00, 33.18it/s]
Loading 0:  95%|█████████▌| 346/363 [00:09<00:00, 40.39it/s]
Loading 0:  97%|█████████▋| 351/363 [00:09<00:00, 40.89it/s]
Loading 0:  98%|█████████▊| 356/363 [00:09<00:00, 41.38it/s]
Loading 0:  99%|█████████▉| 361/363 [00:09<00:00, 43.03it/s]
                                                            

Job mistralai-mistral-nem-93303-v500-mkmlizer completed after 104.75s with status: succeeded
Stopping job with name mistralai-mistral-nem-93303-v500-mkmlizer
Pipeline stage MKMLizer completed in 105.32s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-nem-93303-v500
Waiting for inference service mistralai-mistral-nem-93303-v500 to be ready
Inference service mistralai-mistral-nem-93303-v500 ready after 161.15698862075806s
Pipeline stage MKMLDeployer completed in 161.76s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3428728580474854s
Received healthy response to inference request in 1.6283304691314697s
Received healthy response to inference request in 1.5471258163452148s
Received healthy response to inference request in 1.7129309177398682s
Received healthy response to inference request in 1.5589532852172852s
5 requests
0 failed requests
5th percentile: 1.5494913101196288
10th percentile: 1.551856803894043
20th percentile: 1.5565877914428712
30th percentile: 1.572828722000122
40th percentile: 1.600579595565796
50th percentile: 1.6283304691314697
60th percentile: 1.662170648574829
70th percentile: 1.6960108280181885
80th percentile: 1.8389193058013917
90th percentile: 2.0908960819244387
95th percentile: 2.216884469985962
99th percentile: 2.3176751804351805
mean time: 1.7580426692962647
Pipeline stage StressChecker completed in 10.09s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.70s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.13s
Shutdown handler de-registered
mistralai-mistral-nem_93303_v500 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3038.82s
Shutdown handler de-registered
mistralai-mistral-nem_93303_v500 status is now inactive due to auto deactivation removed underperforming models
mistralai-mistral-nem_93303_v500 status is now torndown due to DeploymentManager action