submission_id: mistralai-mistral-nemo_9330_v194
developer_uid: chai_backend_admin
best_of: 8
celo_rating: 1245.74
display_name: mistralai-mistral-nemo_9330_v194
family_friendly_score: 0.5786
family_friendly_standard_error: 0.006983151723971061
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
gpu_counts: {'NVIDIA RTX A5000': 1}
is_internal_developer: True
language_model: mistralai/Mistral-Nemo-Instruct-2407
latencies: [{'batch_size': 1, 'throughput': 0.6041004047502698, 'latency_mean': 1.655260179042816, 'latency_p50': 1.6519267559051514, 'latency_p90': 1.8323219776153565}, {'batch_size': 3, 'throughput': 1.0961353149805644, 'latency_mean': 2.730257365703583, 'latency_p50': 2.7320470809936523, 'latency_p90': 3.0252765893936155}, {'batch_size': 5, 'throughput': 1.318909243505441, 'latency_mean': 3.778353453874588, 'latency_p50': 3.7895617485046387, 'latency_p90': 4.156122040748596}, {'batch_size': 6, 'throughput': 1.3830696232373454, 'latency_mean': 4.312871580123901, 'latency_p50': 4.331161856651306, 'latency_p90': 4.842809176445007}, {'batch_size': 8, 'throughput': 1.4494390688593624, 'latency_mean': 5.491524139642715, 'latency_p50': 5.480210900306702, 'latency_p90': 6.211421036720276}, {'batch_size': 10, 'throughput': 1.4746468523793508, 'latency_mean': 6.748433440923691, 'latency_p50': 6.695248126983643, 'latency_p90': 7.730241012573242}]
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: mistralai/Mistral-Nemo-I
model_name: mistralai-mistral-nemo_9330_v194
model_num_parameters: 12772070400.0
model_repo: mistralai/Mistral-Nemo-Instruct-2407
model_size: 13B
num_battles: 18216
num_wins: 9226
ranking_group: single
status: inactive
submission_type: basic
throughput_3p7s: 1.31
timestamp: 2024-11-06T16:53:51+00:00
us_pacific_date: 2024-11-06
win_ratio: 0.506477821695213
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nemo-9330-v194-mkmlizer
Waiting for job on mistralai-mistral-nemo-9330-v194-mkmlizer to finish
mistralai-mistral-nemo-9330-v194-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-nemo-9330-v194-mkmlizer: ║     _____            __           __                                ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║       /___/                                                         ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  Version: 0.11.28                                                   ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  belonging to:                                                      ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v194-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
mistralai-mistral-nemo-9330-v194-mkmlizer: Downloaded to shared memory in 55.643s
mistralai-mistral-nemo-9330-v194-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmplwe931se, device:0
mistralai-mistral-nemo-9330-v194-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nemo-9330-v194-mkmlizer: quantized model in 36.855s
mistralai-mistral-nemo-9330-v194-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 92.498s
mistralai-mistral-nemo-9330-v194-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nemo-9330-v194-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nemo-9330-v194-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v194
mistralai-mistral-nemo-9330-v194-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v194/config.json
mistralai-mistral-nemo-9330-v194-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v194/special_tokens_map.json
mistralai-mistral-nemo-9330-v194-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v194/tokenizer_config.json
mistralai-mistral-nemo-9330-v194-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v194/tokenizer.json
mistralai-mistral-nemo-9330-v194-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v194/flywheel_model.0.safetensors
mistralai-mistral-nemo-9330-v194-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 29.03it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 48.56it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:09, 37.83it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:09, 36.91it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:07, 43.20it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 41.99it/s]
Loading 0:  12%|█▏        | 42/363 [00:01<00:07, 41.17it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 46.63it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 44.47it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 33.99it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:09, 32.66it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:07, 38.08it/s]
Loading 0:  21%|██        | 76/363 [00:01<00:07, 36.53it/s]
Loading 0:  22%|██▏       | 80/363 [00:02<00:07, 36.86it/s]
Loading 0:  23%|██▎       | 84/363 [00:02<00:07, 35.60it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:07, 37.69it/s]
Loading 0:  26%|██▌       | 93/363 [00:02<00:07, 36.83it/s]
Loading 0:  27%|██▋       | 98/363 [00:02<00:06, 39.62it/s]
Loading 0:  28%|██▊       | 103/363 [00:02<00:06, 40.28it/s]
Loading 0:  30%|███       | 110/363 [00:02<00:06, 41.81it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 42.36it/s]
Loading 0:  33%|███▎      | 120/363 [00:03<00:06, 39.32it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:05, 41.40it/s]
Loading 0:  36%|███▌      | 130/363 [00:03<00:05, 40.87it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 40.75it/s]
Loading 0:  39%|███▊      | 140/363 [00:03<00:05, 42.84it/s]
Loading 0:  40%|███▉      | 145/363 [00:03<00:08, 26.59it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:07, 26.77it/s]
Loading 0:  43%|████▎     | 156/363 [00:04<00:05, 34.54it/s]
Loading 0:  44%|████▍     | 161/363 [00:04<00:05, 36.79it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:05, 38.89it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:04, 38.97it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 38.72it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:04, 44.02it/s]
Loading 0:  52%|█████▏    | 189/363 [00:04<00:03, 44.98it/s]
Loading 0:  53%|█████▎    | 194/363 [00:05<00:04, 37.73it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:03, 44.70it/s]
Loading 0:  57%|█████▋    | 206/363 [00:05<00:03, 44.62it/s]
Loading 0:  58%|█████▊    | 211/363 [00:05<00:03, 44.54it/s]
Loading 0:  60%|█████▉    | 216/363 [00:05<00:03, 45.31it/s]
Loading 0:  61%|██████    | 222/363 [00:05<00:03, 41.99it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:04, 31.61it/s]
Loading 0:  64%|██████▎   | 231/363 [00:06<00:04, 31.85it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 37.88it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 40.16it/s]
Loading 0:  68%|██████▊   | 248/363 [00:06<00:02, 38.65it/s]
Loading 0:  71%|███████   | 256/363 [00:06<00:02, 46.71it/s]
Loading 0:  72%|███████▏  | 262/363 [00:06<00:02, 45.95it/s]
Loading 0:  74%|███████▎  | 267/363 [00:06<00:02, 44.77it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:01, 49.18it/s]
Loading 0:  77%|███████▋  | 280/363 [00:07<00:01, 43.42it/s]
Loading 0:  79%|███████▊  | 285/363 [00:07<00:01, 42.64it/s]
Loading 0:  80%|████████  | 292/363 [00:07<00:01, 47.29it/s]
Loading 0:  82%|████████▏ | 298/363 [00:07<00:01, 45.72it/s]
Loading 0:  84%|████████▎ | 304/363 [00:14<00:21,  2.78it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:15,  3.48it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:11,  4.44it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:05,  7.18it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:03,  9.63it/s]
Loading 0:  91%|█████████ | 331/363 [00:15<00:02, 12.15it/s]
Loading 0:  93%|█████████▎| 337/363 [00:15<00:01, 16.00it/s]
Loading 0:  94%|█████████▍| 342/363 [00:15<00:01, 19.54it/s]
Loading 0:  96%|█████████▌| 347/363 [00:15<00:00, 23.22it/s]
Loading 0:  97%|█████████▋| 352/363 [00:15<00:00, 27.21it/s]
Loading 0:  98%|█████████▊| 357/363 [00:15<00:00, 26.82it/s]
                                                            

Job mistralai-mistral-nemo-9330-v194-mkmlizer completed after 114.06s with status: succeeded
Stopping job with name mistralai-mistral-nemo-9330-v194-mkmlizer
Pipeline stage MKMLizer completed in 114.56s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-nemo-9330-v194
Waiting for inference service mistralai-mistral-nemo-9330-v194 to be ready
Inference service mistralai-mistral-nemo-9330-v194 ready after 170.93791246414185s
Pipeline stage MKMLDeployer completed in 171.70s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1158227920532227s
Received healthy response to inference request in 2.032949924468994s
Received healthy response to inference request in 1.4600558280944824s
Received healthy response to inference request in 1.7532227039337158s
Failed to get response for submission bbchicago-nana-nemo-12b-v1-0_v8: ('http://bbchicago-nana-nemo-12b-v1-0-v8-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Received healthy response to inference request in 1.645522117614746s
5 requests
0 failed requests
5th percentile: 1.4971490859985352
10th percentile: 1.5342423439025878
20th percentile: 1.6084288597106933
30th percentile: 1.66706223487854
40th percentile: 1.710142469406128
50th percentile: 1.7532227039337158
60th percentile: 1.8651135921478272
70th percentile: 1.9770044803619384
80th percentile: 2.04952449798584
90th percentile: 2.0826736450195313
95th percentile: 2.0992482185363768
99th percentile: 2.1125078773498536
mean time: 1.8015146732330323
Pipeline stage StressChecker completed in 10.65s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.51s
Shutdown handler de-registered
mistralai-mistral-nemo_9330_v194 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2765.90s
Shutdown handler de-registered
mistralai-mistral-nemo_9330_v194 status is now inactive due to auto deactivation removed underperforming models