submission_id: mistralai-mistral-nemo-_9330_v59
developer_uid: albert_chai
alignment_samples: 10160
alignment_score: -1.0374091660623148
best_of: 8
celo_rating: 1238.85
display_name: nemo-dpo-bo8x1024_T12
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.2, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
is_internal_developer: True
language_model: mistralai/Mistral-Nemo-Instruct-2407
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: mistralai/Mistral-Nemo-I
model_name: nemo-dpo-bo8x1024_T12
model_num_parameters: 12772070400.0
model_repo: mistralai/Mistral-Nemo-Instruct-2407
model_size: 13B
num_battles: 10160
num_wins: 5128
propriety_score: 0.7116022099447514
propriety_total_count: 905.0
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-08-24T05:54:53+00:00
us_pacific_date: 2024-08-23
win_ratio: 0.5047244094488189
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nemo-9330-v59-mkmlizer
Waiting for job on mistralai-mistral-nemo-9330-v59-mkmlizer to finish
mistralai-mistral-nemo-9330-v59-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-nemo-9330-v59-mkmlizer: ║     _____            __           __                                ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║       /___/                                                         ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  Version: 0.10.1                                                    ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  belonging to:                                                      ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v59-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
mistralai-mistral-nemo-9330-v59-mkmlizer: Downloaded to shared memory in 48.787s
mistralai-mistral-nemo-9330-v59-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpghf7dsvw, device:0
mistralai-mistral-nemo-9330-v59-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nemo-9330-v59-mkmlizer: quantized model in 37.561s
mistralai-mistral-nemo-9330-v59-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 86.348s
mistralai-mistral-nemo-9330-v59-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nemo-9330-v59-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nemo-9330-v59-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v59
mistralai-mistral-nemo-9330-v59-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v59/config.json
mistralai-mistral-nemo-9330-v59-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v59/special_tokens_map.json
mistralai-mistral-nemo-9330-v59-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v59/tokenizer_config.json
mistralai-mistral-nemo-9330-v59-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v59/tokenizer.json
mistralai-mistral-nemo-9330-v59-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v59/flywheel_model.0.safetensors
mistralai-mistral-nemo-9330-v59-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.00it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:07, 47.44it/s]
Loading 0:   5%|▍         | 18/363 [00:00<00:07, 48.59it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 39.52it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:07, 45.17it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 41.86it/s]
Loading 0:  12%|█▏        | 42/363 [00:01<00:07, 40.59it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:06, 45.30it/s]
Loading 0:  15%|█▍        | 53/363 [00:01<00:06, 44.73it/s]
Loading 0:  16%|█▌        | 58/363 [00:01<00:06, 45.17it/s]
Loading 0:  17%|█▋        | 63/363 [00:01<00:09, 30.11it/s]
Loading 0:  19%|█▉        | 69/363 [00:01<00:09, 32.19it/s]
Loading 0:  20%|██        | 73/363 [00:01<00:09, 31.45it/s]
Loading 0:  22%|██▏       | 80/363 [00:02<00:07, 38.49it/s]
Loading 0:  23%|██▎       | 85/363 [00:02<00:07, 38.40it/s]
Loading 0:  25%|██▍       | 90/363 [00:02<00:07, 38.11it/s]
Loading 0:  26%|██▌       | 95/363 [00:02<00:06, 40.54it/s]
Loading 0:  28%|██▊       | 100/363 [00:02<00:07, 34.34it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:06, 38.82it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 42.88it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 41.16it/s]
Loading 0:  34%|███▎      | 122/363 [00:03<00:05, 42.68it/s]
Loading 0:  35%|███▍      | 127/363 [00:03<00:06, 35.12it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:05, 41.62it/s]
Loading 0:  38%|███▊      | 139/363 [00:03<00:05, 41.87it/s]
Loading 0:  40%|███▉      | 144/363 [00:03<00:08, 25.82it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:07, 28.03it/s]
Loading 0:  43%|████▎     | 156/363 [00:04<00:06, 34.44it/s]
Loading 0:  44%|████▍     | 161/363 [00:04<00:05, 35.71it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:05, 37.21it/s]
Loading 0:  47%|████▋     | 171/363 [00:04<00:04, 39.69it/s]
Loading 0:  48%|████▊     | 176/363 [00:04<00:05, 32.90it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 39.51it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:04, 39.60it/s]
Loading 0:  53%|█████▎    | 193/363 [00:05<00:04, 39.86it/s]
Loading 0:  55%|█████▍    | 198/363 [00:05<00:03, 41.78it/s]
Loading 0:  56%|█████▌    | 203/363 [00:05<00:04, 34.17it/s]
Loading 0:  57%|█████▋    | 208/363 [00:05<00:04, 37.62it/s]
Loading 0:  59%|█████▊    | 213/363 [00:05<00:04, 36.03it/s]
Loading 0:  60%|██████    | 218/363 [00:05<00:03, 37.39it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:04, 28.67it/s]
Loading 0:  63%|██████▎   | 227/363 [00:06<00:04, 29.50it/s]
Loading 0:  64%|██████▎   | 231/363 [00:06<00:04, 28.74it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 34.18it/s]
Loading 0:  66%|██████▋   | 241/363 [00:06<00:03, 34.19it/s]
Loading 0:  68%|██████▊   | 246/363 [00:06<00:03, 37.46it/s]
Loading 0:  69%|██████▉   | 250/363 [00:06<00:03, 36.64it/s]
Loading 0:  70%|███████   | 255/363 [00:06<00:02, 38.11it/s]
Loading 0:  71%|███████▏  | 259/363 [00:07<00:02, 37.09it/s]
Loading 0:  73%|███████▎  | 264/363 [00:07<00:02, 39.60it/s]
Loading 0:  74%|███████▍  | 269/363 [00:07<00:02, 39.50it/s]
Loading 0:  75%|███████▌  | 273/363 [00:07<00:02, 39.24it/s]
Loading 0:  76%|███████▋  | 277/363 [00:07<00:02, 35.90it/s]
Loading 0:  78%|███████▊  | 282/363 [00:07<00:02, 37.74it/s]
Loading 0:  79%|███████▉  | 286/363 [00:07<00:02, 36.79it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 39.13it/s]
Loading 0:  81%|████████▏ | 295/363 [00:07<00:01, 38.14it/s]
Loading 0:  82%|████████▏ | 299/363 [00:08<00:01, 36.63it/s]
Loading 0:  84%|████████▎ | 304/363 [00:14<00:28,  2.09it/s]
Loading 0:  85%|████████▍ | 307/363 [00:15<00:21,  2.63it/s]
Loading 0:  86%|████████▌ | 312/363 [00:15<00:13,  3.89it/s]
Loading 0:  88%|████████▊ | 320/363 [00:15<00:06,  6.66it/s]
Loading 0:  90%|████████▉ | 325/363 [00:15<00:04,  8.82it/s]
Loading 0:  91%|█████████ | 330/363 [00:15<00:03, 10.74it/s]
Loading 0:  93%|█████████▎| 336/363 [00:15<00:01, 14.74it/s]
Loading 0:  94%|█████████▍| 341/363 [00:15<00:01, 17.81it/s]
Loading 0:  95%|█████████▌| 346/363 [00:15<00:00, 21.78it/s]
Loading 0:  97%|█████████▋| 351/363 [00:16<00:00, 25.55it/s]
Loading 0:  98%|█████████▊| 356/363 [00:16<00:00, 28.60it/s]
Loading 0:  99%|█████████▉| 361/363 [00:16<00:00, 31.87it/s]
                                                            

Job mistralai-mistral-nemo-9330-v59-mkmlizer completed after 128.93s with status: succeeded
Stopping job with name mistralai-mistral-nemo-9330-v59-mkmlizer
Pipeline stage MKMLizer completed in 129.96s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.10s
Running pipeline stage ISVCDeployer
Creating inference service mistralai-mistral-nemo-9330-v59
Waiting for inference service mistralai-mistral-nemo-9330-v59 to be ready
Inference service mistralai-mistral-nemo-9330-v59 ready after 232.4188244342804s
Pipeline stage ISVCDeployer completed in 233.24s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.4586868286132812s
Received healthy response to inference request in 1.751861333847046s
Received healthy response to inference request in 1.6973342895507812s
Received healthy response to inference request in 1.7602057456970215s
Received healthy response to inference request in 1.971355676651001s
5 requests
0 failed requests
5th percentile: 1.5064163208007812
10th percentile: 1.5541458129882812
20th percentile: 1.6496047973632812
30th percentile: 1.708239698410034
40th percentile: 1.73005051612854
50th percentile: 1.751861333847046
60th percentile: 1.755199098587036
70th percentile: 1.7585368633270264
80th percentile: 1.8024357318878175
90th percentile: 1.8868957042694092
95th percentile: 1.929125690460205
99th percentile: 1.9629096794128418
mean time: 1.7278887748718261
Pipeline stage StressChecker completed in 9.34s
mistralai-mistral-nemo-_9330_v59 status is now deployed due to DeploymentManager action
mistralai-mistral-nemo-_9330_v59 status is now inactive due to auto deactivation removed underperforming models
mistralai-mistral-nemo-_9330_v59 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics