mistralai-mistral-nemo_9330

submission_id: mistralai-mistral-nemo_9330_v171

developer_uid: chai_backend_admin

best_of: 8

celo_rating: 1244.05

display_name: mistralai-mistral-nemo_9330_v171

family_friendly_score: 0.5929110653586855

family_friendly_standard_error: 0.0053789459040357816

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 50, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

is_internal_developer: True

language_model: mistralai/Mistral-Nemo-Instruct-2407

max_input_tokens: 1024

max_output_tokens: 64

model_architecture: MistralForCausalLM

model_group: mistralai/Mistral-Nemo-I

model_name: mistralai-mistral-nemo_9330_v171

model_num_parameters: 12772070400.0

model_repo: mistralai/Mistral-Nemo-Instruct-2407

model_size: 13B

num_battles: 8707

num_wins: 4259

ranking_group: single

status: inactive

submission_type: basic

timestamp: 2024-10-18T14:35:35+00:00

us_pacific_date: 2024-10-18

win_ratio: 0.4891466636039968

Download Preference Data

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nemo-9330-v171-mkmlizer
Waiting for job on mistralai-mistral-nemo-9330-v171-mkmlizer to finish
mistralai-mistral-nemo-9330-v171-mkmlizer: Downloaded to shared memory in 49.467s
mistralai-mistral-nemo-9330-v171-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpb_fl6yw8, device:0
mistralai-mistral-nemo-9330-v171-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nemo-9330-v171-mkmlizer: quantized model in 35.696s
mistralai-mistral-nemo-9330-v171-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 85.163s
mistralai-mistral-nemo-9330-v171-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nemo-9330-v171-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nemo-9330-v171-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v171
mistralai-mistral-nemo-9330-v171-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v171/config.json
mistralai-mistral-nemo-9330-v171-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v171/special_tokens_map.json
mistralai-mistral-nemo-9330-v171-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v171/tokenizer_config.json
mistralai-mistral-nemo-9330-v171-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v171/tokenizer.json
mistralai-mistral-nemo-9330-v171-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v171/flywheel_model.0.safetensors
mistralai-mistral-nemo-9330-v171-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 30.71it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 50.89it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 46.28it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 44.58it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 50.42it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 43.16it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 42.15it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 48.18it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 46.89it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 34.68it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:08, 35.70it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 40.46it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:07, 39.54it/s]
Loading 0:  23%|██▎       | 83/363 [00:02<00:07, 38.75it/s]
Loading 0:  25%|██▍       | 90/363 [00:02<00:06, 43.22it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:06, 42.61it/s]
Loading 0:  28%|██▊       | 101/363 [00:02<00:06, 41.95it/s]
Loading 0:  30%|██▉       | 108/363 [00:02<00:05, 48.63it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:05, 45.67it/s]
Loading 0:  33%|███▎      | 119/363 [00:02<00:05, 44.59it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:04, 48.28it/s]
Loading 0:  36%|███▌      | 131/363 [00:02<00:04, 48.60it/s]
Loading 0:  37%|███▋      | 136/363 [00:03<00:05, 40.11it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:06, 33.70it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 33.75it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 32.75it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 38.88it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:04, 41.03it/s]
Loading 0:  46%|████▌     | 166/363 [00:03<00:04, 42.33it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:04, 41.10it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 41.15it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:03, 45.86it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:03, 43.83it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 44.59it/s]
Loading 0:  55%|█████▍    | 199/363 [00:04<00:03, 42.94it/s]
Loading 0:  56%|█████▌    | 204/363 [00:04<00:03, 42.90it/s]
Loading 0:  58%|█████▊    | 210/363 [00:04<00:03, 46.61it/s]
Loading 0:  59%|█████▉    | 215/363 [00:05<00:03, 44.11it/s]
Loading 0:  61%|██████    | 220/363 [00:05<00:03, 44.46it/s]
Loading 0:  62%|██████▏   | 225/363 [00:05<00:05, 27.37it/s]
Loading 0:  63%|██████▎   | 230/363 [00:05<00:04, 29.84it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 37.83it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:03, 39.04it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:02, 40.11it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 42.02it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:02, 35.60it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 41.96it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 42.77it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 43.76it/s]
Loading 0:  77%|███████▋  | 280/363 [00:06<00:01, 43.15it/s]
Loading 0:  79%|███████▊  | 285/363 [00:06<00:01, 42.48it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 46.44it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 45.71it/s]
Loading 0:  83%|████████▎ | 303/363 [00:07<00:01, 45.97it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:20,  2.65it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:15,  3.38it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:07,  5.48it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:04,  7.44it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:03,  9.51it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 13.52it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:01, 17.12it/s]
Loading 0:  96%|█████████▋| 350/363 [00:14<00:00, 20.78it/s]
Loading 0:  98%|█████████▊| 355/363 [00:15<00:00, 24.45it/s]
Loading 0:  99%|█████████▉| 360/363 [00:15<00:00, 27.83it/s]
                                                            
Job mistralai-mistral-nemo-9330-v171-mkmlizer completed after 113.03s with status: succeeded
Stopping job with name mistralai-mistral-nemo-9330-v171-mkmlizer
Pipeline stage MKMLizer completed in 114.01s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-nemo-9330-v171
Waiting for inference service mistralai-mistral-nemo-9330-v171 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service mistralai-mistral-nemo-9330-v171 ready after 180.968323469162s
Pipeline stage MKMLDeployer completed in 181.46s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.618941307067871s
Received healthy response to inference request in 1.221085786819458s
Received healthy response to inference request in 1.735046148300171s
Received healthy response to inference request in 1.8059437274932861s
Received healthy response to inference request in 1.3287792205810547s
5 requests
0 failed requests
5th percentile: 1.2426244735717773
10th percentile: 1.2641631603240966
20th percentile: 1.3072405338287354
30th percentile: 1.410032606124878
40th percentile: 1.5725393772125245
50th percentile: 1.735046148300171
60th percentile: 1.763405179977417
70th percentile: 1.791764211654663
80th percentile: 1.9685432434082033
90th percentile: 2.293742275238037
95th percentile: 2.456341791152954
99th percentile: 2.586421403884888
mean time: 1.741959238052368
Pipeline stage StressChecker completed in 10.24s
Shutdown handler de-registered
mistralai-mistral-nemo_9330_v171 status is now deployed due to DeploymentManager action
mistralai-mistral-nemo_9330_v171 status is now inactive due to auto deactivation removed underperforming models