mistralai-mistral-nemo-_9330

developer_uid: azuruce

submission_id: mistralai-mistral-nemo-_9330_v81

model_name: ebony-horror-baseline2

model_group: mistralai/Mistral-Nemo-I

status: torndown

timestamp: 2024-09-17T06:26:08+00:00

num_battles: 10341

num_wins: 4616

celo_rating: 1212.55

family_friendly_score: 0.0

submission_type: basic

model_repo: mistralai/Mistral-Nemo-Instruct-2407

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 4

max_input_tokens: 1024

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.633603092957823, 'latency_mean': 1.5781825351715089, 'latency_p50': 1.5583665370941162, 'latency_p90': 1.7428905248641968}, {'batch_size': 3, 'throughput': 1.2293181353847038, 'latency_mean': 2.4342538392543793, 'latency_p50': 2.4367562532424927, 'latency_p90': 2.670191717147827}, {'batch_size': 5, 'throughput': 1.5332758664635815, 'latency_mean': 3.2444242632389066, 'latency_p50': 3.217241048812866, 'latency_p90': 3.672373652458191}, {'batch_size': 6, 'throughput': 1.5977581748167862, 'latency_mean': 3.7362730753421785, 'latency_p50': 3.7161799669265747, 'latency_p90': 4.2362206220626835}, {'batch_size': 8, 'throughput': 1.7094263196948218, 'latency_mean': 4.654526592493057, 'latency_p50': 4.656731128692627, 'latency_p90': 5.169000625610352}, {'batch_size': 10, 'throughput': 1.756127770221349, 'latency_mean': 5.6589939916133885, 'latency_p50': 5.705182433128357, 'latency_p90': 6.4868361234664915}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: ebony-horror-baseline2

is_internal_developer: True

language_model: mistralai/Mistral-Nemo-Instruct-2407

model_size: 13B

ranking_group: single

throughput_3p7s: 1.6

us_pacific_date: 2024-09-16

win_ratio: 0.4463784933758824

generation_params: {'temperature': 0.9, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.5, 'frequency_penalty': 0.5, 'stopping_words': ['\n', '</s>'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n##Try to make the conversation scary while stay in character##\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nemo-9330-v81-mkmlizer
Waiting for job on mistralai-mistral-nemo-9330-v81-mkmlizer to finish
mistralai-mistral-nemo-9330-v81-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-nemo-9330-v81-mkmlizer: ║     _____            __           __                                ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║       /___/                                                         ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  Version: 0.10.1                                                    ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  belonging to:                                                      ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v81-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
mistralai-mistral-nemo-9330-v81-mkmlizer: Downloaded to shared memory in 45.778s
mistralai-mistral-nemo-9330-v81-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp62y8jluw, device:0
mistralai-mistral-nemo-9330-v81-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nemo-9330-v81-mkmlizer: quantized model in 35.714s
mistralai-mistral-nemo-9330-v81-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 81.492s
mistralai-mistral-nemo-9330-v81-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nemo-9330-v81-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nemo-9330-v81-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v81
mistralai-mistral-nemo-9330-v81-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v81/config.json
mistralai-mistral-nemo-9330-v81-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v81/special_tokens_map.json
mistralai-mistral-nemo-9330-v81-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v81/tokenizer_config.json
mistralai-mistral-nemo-9330-v81-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v81/tokenizer.json
mistralai-mistral-nemo-9330-v81-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v81/flywheel_model.0.safetensors
mistralai-mistral-nemo-9330-v81-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:10, 34.11it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:07, 46.61it/s]
Loading 0:   6%|▋         | 23/363 [00:00<00:06, 49.98it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:05, 58.39it/s]
Loading 0:  10%|█         | 38/363 [00:00<00:06, 52.65it/s]
Loading 0:  12%|█▏        | 44/363 [00:00<00:06, 52.24it/s]
Loading 0:  14%|█▍        | 50/363 [00:01<00:07, 44.44it/s]
Loading 0:  17%|█▋        | 60/363 [00:01<00:05, 52.09it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:07, 37.48it/s]
Loading 0:  20%|██        | 73/363 [00:01<00:07, 38.96it/s]
Loading 0:  22%|██▏       | 81/363 [00:01<00:06, 46.09it/s]
Loading 0:  24%|██▍       | 87/363 [00:01<00:06, 44.62it/s]
Loading 0:  25%|██▌       | 92/363 [00:02<00:06, 42.87it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:05, 47.52it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:05, 44.24it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 47.40it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 46.51it/s]
Loading 0:  34%|███▎      | 122/363 [00:02<00:05, 45.26it/s]
Loading 0:  35%|███▍      | 127/363 [00:02<00:06, 37.89it/s]
Loading 0:  37%|███▋      | 134/363 [00:02<00:05, 42.83it/s]
Loading 0:  38%|███▊      | 139/363 [00:03<00:05, 44.04it/s]
Loading 0:  40%|███▉      | 144/363 [00:03<00:07, 29.02it/s]
Loading 0:  41%|████      | 149/363 [00:03<00:06, 30.90it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 38.45it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:05, 39.95it/s]
Loading 0:  46%|████▌     | 167/363 [00:03<00:05, 38.66it/s]
Loading 0:  48%|████▊     | 175/363 [00:04<00:04, 46.53it/s]
Loading 0:  50%|████▉     | 181/363 [00:04<00:04, 45.35it/s]
Loading 0:  51%|█████     | 186/363 [00:04<00:03, 44.74it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 49.47it/s]
Loading 0:  55%|█████▍    | 199/363 [00:04<00:03, 44.02it/s]
Loading 0:  56%|█████▌    | 204/363 [00:04<00:03, 41.62it/s]
Loading 0:  58%|█████▊    | 211/363 [00:04<00:03, 46.72it/s]
Loading 0:  60%|█████▉    | 217/363 [00:04<00:03, 46.29it/s]
Loading 0:  61%|██████    | 222/363 [00:05<00:03, 45.75it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:04, 32.70it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:04, 31.84it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 37.21it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:03, 39.77it/s]
Loading 0:  68%|██████▊   | 247/363 [00:05<00:02, 41.66it/s]
Loading 0:  69%|██████▉   | 252/363 [00:05<00:02, 42.45it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:03, 33.51it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 40.25it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 39.50it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 41.66it/s]
Loading 0:  77%|███████▋  | 280/363 [00:06<00:01, 42.06it/s]
Loading 0:  79%|███████▊  | 285/363 [00:06<00:01, 41.10it/s]
Loading 0:  80%|████████  | 291/363 [00:06<00:01, 44.40it/s]
Loading 0:  82%|████████▏ | 296/363 [00:06<00:01, 44.66it/s]
Loading 0:  83%|████████▎ | 303/363 [00:07<00:01, 45.73it/s]
Loading 0:  85%|████████▍ | 308/363 [00:13<00:20,  2.66it/s]
Loading 0:  86%|████████▌ | 312/363 [00:13<00:15,  3.39it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:07,  5.49it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:04,  7.45it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:03,  9.49it/s]
Loading 0:  93%|█████████▎| 339/363 [00:14<00:01, 13.66it/s]
Loading 0:  96%|█████████▌| 348/363 [00:14<00:00, 19.06it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 25.26it/s]
Loading 0: 100%|█████████▉| 362/363 [00:14<00:00, 28.76it/s]
                                                            
Job mistralai-mistral-nemo-9330-v81-mkmlizer completed after 103.27s with status: succeeded
Stopping job with name mistralai-mistral-nemo-9330-v81-mkmlizer
Pipeline stage MKMLizer completed in 103.88s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.08s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-nemo-9330-v81
Waiting for inference service mistralai-mistral-nemo-9330-v81 to be ready
Inference service mistralai-mistral-nemo-9330-v81 ready after 170.49850702285767s
Pipeline stage MKMLDeployer completed in 170.84s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9887113571166992s
Received healthy response to inference request in 2.4025766849517822s
Received healthy response to inference request in 1.2537620067596436s
Received healthy response to inference request in 1.6513173580169678s
Received healthy response to inference request in 1.6967103481292725s
5 requests
0 failed requests
5th percentile: 1.3332730770111083
10th percentile: 1.4127841472625733
20th percentile: 1.571806287765503
30th percentile: 1.6603959560394288
40th percentile: 1.6785531520843506
50th percentile: 1.6967103481292725
60th percentile: 1.8135107517242433
70th percentile: 1.9303111553192138
80th percentile: 2.071484422683716
90th percentile: 2.237030553817749
95th percentile: 2.3198036193847655
99th percentile: 2.386022071838379
mean time: 1.7986155509948731
Pipeline stage StressChecker completed in 9.58s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.76s
Shutdown handler de-registered
mistralai-mistral-nemo-_9330_v81 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service mistralai-mistral-nemo-9330-v81-profiler
Waiting for inference service mistralai-mistral-nemo-9330-v81-profiler to be ready
Inference service mistralai-mistral-nemo-9330-v81-profiler ready after 170.42317485809326s
Pipeline stage MKMLProfilerDeployer completed in 170.78s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/mistralai-mistral-neae9db02a7d9d6df82cf2265cdea4e1cc-deplorhs4j:/code/chaiverse_profiler_1726554869 --namespace tenant-chaiml-guanaco
kubectl exec -it mistralai-mistral-neae9db02a7d9d6df82cf2265cdea4e1cc-deplorhs4j --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726554869 && python profiles.py profile --best_of_n 4 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1726554869/summary.json'
kubectl exec -it mistralai-mistral-neae9db02a7d9d6df82cf2265cdea4e1cc-deplorhs4j --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726554869/summary.json'
Pipeline stage MKMLProfilerRunner completed in 971.35s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service mistralai-mistral-nemo-9330-v81-profiler is running
Tearing down inference service mistralai-mistral-nemo-9330-v81-profiler
Service mistralai-mistral-nemo-9330-v81-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.96s
Shutdown handler de-registered
mistralai-mistral-nemo-_9330_v81 status is now inactive due to auto deactivation removed underperforming models
mistralai-mistral-nemo-_9330_v81 status is now torndown due to DeploymentManager action