mistralai-mistral-small_5341

developer_uid: azuruce

submission_id: mistralai-mistral-small_5341_v91

model_name: small-prompt

model_group: mistralai/Mistral-Small-

status: inactive

timestamp: 2024-11-20T14:23:21+00:00

num_battles: 13095

num_wins: 5975

celo_rating: 1220.68

family_friendly_score: 0.5786

family_friendly_standard_error: 0.006983151723971061

submission_type: basic

model_repo: mistralai/Mistral-Small-Instruct-2409

model_architecture: MistralForCausalLM

model_num_parameters: 22247282688.0

best_of: 4

max_input_tokens: 1024

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.3831278411202959, 'latency_mean': 2.6100097274780274, 'latency_p50': 2.592982292175293, 'latency_p90': 2.8832868337631226}, {'batch_size': 3, 'throughput': 0.8147656364323776, 'latency_mean': 3.6733777809143064, 'latency_p50': 3.6753162145614624, 'latency_p90': 4.001369261741638}, {'batch_size': 5, 'throughput': 1.0885499559975713, 'latency_mean': 4.556945215463639, 'latency_p50': 4.52631676197052, 'latency_p90': 5.017989253997802}, {'batch_size': 6, 'throughput': 1.17154889504076, 'latency_mean': 5.1042794156074525, 'latency_p50': 5.082825422286987, 'latency_p90': 5.591396307945251}, {'batch_size': 10, 'throughput': 1.386536527948177, 'latency_mean': 7.146538611650467, 'latency_p50': 7.183301687240601, 'latency_p90': 7.955365991592407}]

gpu_counts: {'NVIDIA RTX A6000': 1}

display_name: small-prompt

is_internal_developer: True

language_model: mistralai/Mistral-Small-Instruct-2409

model_size: 22B

ranking_group: single

throughput_3p7s: 0.83

us_pacific_date: 2024-11-20

win_ratio: 0.4562810232913326

generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}

formatter: {'memory_template': 'In your response, avoid always starting with the subject or entity. Mix it up by beginning with a question, a verb, or other sentence structures. Respond in detail to deliver vivid and long response', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-small-5341-v91-mkmlizer
Waiting for job on mistralai-mistral-small-5341-v91-mkmlizer to finish
mistralai-mistral-small-5341-v91-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-small-5341-v91-mkmlizer: ║     _____            __           __                                ║
mistralai-mistral-small-5341-v91-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
mistralai-mistral-small-5341-v91-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
mistralai-mistral-small-5341-v91-mkmlizer: ║       /___/                                                         ║
mistralai-mistral-small-5341-v91-mkmlizer: ║                                                                     ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  Version: 0.11.12                                                   ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mistral-small-5341-v91-mkmlizer: ║                                                                     ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  belonging to:                                                      ║
mistralai-mistral-small-5341-v91-mkmlizer: ║                                                                     ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mistral-small-5341-v91-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
mistralai-mistral-small-5341-v91-mkmlizer: ║                                                                     ║
mistralai-mistral-small-5341-v91-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
mistralai-mistral-small-5341-v91-mkmlizer: Downloaded to shared memory in 94.172s
mistralai-mistral-small-5341-v91-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpiai7bic_, device:0
mistralai-mistral-small-5341-v91-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-small-5341-v91-mkmlizer: quantized model in 43.738s
mistralai-mistral-small-5341-v91-mkmlizer: Processed model mistralai/Mistral-Small-Instruct-2409 in 137.911s
mistralai-mistral-small-5341-v91-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-small-5341-v91-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-small-5341-v91-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-small-5341-v91
mistralai-mistral-small-5341-v91-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-small-5341-v91/config.json
mistralai-mistral-small-5341-v91-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-small-5341-v91/special_tokens_map.json
mistralai-mistral-small-5341-v91-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-small-5341-v91/tokenizer_config.json
mistralai-mistral-small-5341-v91-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/mistralai-mistral-small-5341-v91/tokenizer.model
mistralai-mistral-small-5341-v91-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-small-5341-v91/tokenizer.json
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
mistralai-mistral-small-5341-v91-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-small-5341-v91/flywheel_model.0.safetensors
mistralai-mistral-small-5341-v91-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:19, 25.38it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:12, 40.42it/s]
Loading 0:   3%|▎         | 17/507 [00:00<00:12, 38.71it/s]
Loading 0:   4%|▍         | 22/507 [00:00<00:12, 38.54it/s]
Loading 0:   5%|▌         | 26/507 [00:00<00:12, 37.53it/s]
Loading 0:   6%|▌         | 31/507 [00:00<00:12, 38.11it/s]
Loading 0:   7%|▋         | 36/507 [00:00<00:11, 40.47it/s]
Loading 0:   8%|▊         | 41/507 [00:01<00:13, 33.37it/s]
Loading 0:   9%|▉         | 47/507 [00:01<00:12, 37.12it/s]
Loading 0:  10%|█         | 51/507 [00:01<00:12, 35.83it/s]
Loading 0:  11%|█         | 55/507 [00:01<00:17, 26.27it/s]
Loading 0:  12%|█▏        | 59/507 [00:01<00:15, 28.05it/s]
Loading 0:  12%|█▏        | 63/507 [00:01<00:14, 30.11it/s]
Loading 0:  13%|█▎        | 67/507 [00:02<00:14, 30.94it/s]
Loading 0:  14%|█▍        | 73/507 [00:02<00:12, 33.77it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:12, 35.34it/s]
Loading 0:  17%|█▋        | 85/507 [00:02<00:11, 37.70it/s]
Loading 0:  18%|█▊        | 89/507 [00:02<00:12, 32.35it/s]
Loading 0:  19%|█▊        | 94/507 [00:02<00:11, 35.57it/s]
Loading 0:  19%|█▉        | 98/507 [00:02<00:13, 31.08it/s]
Loading 0:  21%|██        | 104/507 [00:03<00:10, 37.51it/s]
Loading 0:  21%|██▏       | 109/507 [00:03<00:11, 34.53it/s]
Loading 0:  22%|██▏       | 113/507 [00:03<00:14, 26.83it/s]
Loading 0:  23%|██▎       | 117/507 [00:03<00:14, 26.23it/s]
Loading 0:  24%|██▍       | 122/507 [00:03<00:14, 27.39it/s]
Loading 0:  25%|██▌       | 129/507 [00:03<00:11, 34.16it/s]
Loading 0:  26%|██▌       | 133/507 [00:04<00:11, 33.44it/s]
Loading 0:  27%|██▋       | 138/507 [00:04<00:10, 34.86it/s]
Loading 0:  28%|██▊       | 142/507 [00:04<00:10, 34.02it/s]
Loading 0:  29%|██▉       | 147/507 [00:04<00:09, 36.89it/s]
Loading 0:  30%|██▉       | 151/507 [00:04<00:09, 36.28it/s]
Loading 0:  31%|███       | 156/507 [00:04<00:09, 37.82it/s]
Loading 0:  32%|███▏      | 160/507 [00:04<00:09, 36.19it/s]
Loading 0:  32%|███▏      | 164/507 [00:04<00:09, 35.00it/s]
Loading 0:  33%|███▎      | 169/507 [00:05<00:11, 28.75it/s]
Loading 0:  34%|███▍      | 173/507 [00:05<00:11, 29.41it/s]
Loading 0:  35%|███▍      | 177/507 [00:05<00:11, 28.17it/s]
Loading 0:  36%|███▌      | 183/507 [00:05<00:09, 32.76it/s]
Loading 0:  37%|███▋      | 187/507 [00:05<00:09, 32.28it/s]
Loading 0:  38%|███▊      | 192/507 [00:05<00:09, 34.11it/s]
Loading 0:  39%|███▊      | 196/507 [00:05<00:09, 33.14it/s]
Loading 0:  40%|███▉      | 201/507 [00:06<00:08, 36.16it/s]
Loading 0:  40%|████      | 205/507 [00:06<00:08, 35.90it/s]
Loading 0:  41%|████▏     | 210/507 [00:06<00:07, 38.15it/s]
Loading 0:  42%|████▏     | 214/507 [00:06<00:07, 37.04it/s]
Loading 0:  43%|████▎     | 218/507 [00:06<00:07, 36.59it/s]
Loading 0:  44%|████▍     | 222/507 [00:06<00:07, 36.63it/s]
Loading 0:  45%|████▍     | 226/507 [00:06<00:10, 27.45it/s]
Loading 0:  45%|████▌     | 230/507 [00:06<00:10, 27.03it/s]
Loading 0:  47%|████▋     | 237/507 [00:07<00:07, 35.01it/s]
Loading 0:  48%|████▊     | 241/507 [00:07<00:07, 34.97it/s]
Loading 0:  49%|████▊     | 246/507 [00:07<00:07, 37.21it/s]
Loading 0:  49%|████▉     | 250/507 [00:07<00:07, 36.40it/s]
Loading 0:  50%|█████     | 255/507 [00:07<00:06, 38.41it/s]
Loading 0:  51%|█████     | 259/507 [00:07<00:06, 36.31it/s]
Loading 0:  52%|█████▏    | 264/507 [00:07<00:06, 38.37it/s]
Loading 0:  53%|█████▎    | 268/507 [00:07<00:06, 37.67it/s]
Loading 0:  54%|█████▍    | 273/507 [00:08<00:06, 37.39it/s]
Loading 0:  55%|█████▍    | 277/507 [00:08<00:06, 35.05it/s]
Loading 0:  56%|█████▌    | 282/507 [00:08<00:06, 37.48it/s]
Loading 0:  56%|█████▋    | 286/507 [00:08<00:08, 27.22it/s]
Loading 0:  57%|█████▋    | 290/507 [00:08<00:07, 29.55it/s]
Loading 0:  58%|█████▊    | 294/507 [00:08<00:07, 29.48it/s]
Loading 0:  59%|█████▉    | 299/507 [00:23<00:07, 29.48it/s]
Loading 0:  59%|█████▉    | 300/507 [00:23<03:14,  1.07it/s]
Loading 0:  60%|█████▉    | 302/507 [00:23<02:45,  1.24it/s]
Loading 0:  61%|██████    | 307/507 [00:23<01:46,  1.87it/s]
Loading 0:  61%|██████    | 310/507 [00:23<01:22,  2.38it/s]
Loading 0:  62%|██████▏   | 314/507 [00:23<00:57,  3.34it/s]
Loading 0:  63%|██████▎   | 319/507 [00:24<00:38,  4.93it/s]
Loading 0:  64%|██████▎   | 323/507 [00:24<00:27,  6.60it/s]
Loading 0:  64%|██████▍   | 327/507 [00:24<00:20,  8.65it/s]
Loading 0:  65%|██████▌   | 331/507 [00:24<00:16, 10.98it/s]
Loading 0:  66%|██████▌   | 335/507 [00:24<00:12, 13.40it/s]
Loading 0:  67%|██████▋   | 340/507 [00:24<00:10, 15.73it/s]
Loading 0:  68%|██████▊   | 343/507 [00:24<00:09, 17.27it/s]
Loading 0:  68%|██████▊   | 347/507 [00:24<00:08, 19.37it/s]
Loading 0:  70%|██████▉   | 354/507 [00:25<00:05, 26.84it/s]
Loading 0:  71%|███████   | 358/507 [00:25<00:05, 28.29it/s]
Loading 0:  72%|███████▏  | 363/507 [00:25<00:04, 31.80it/s]
Loading 0:  72%|███████▏  | 367/507 [00:25<00:04, 32.29it/s]
Loading 0:  73%|███████▎  | 372/507 [00:25<00:03, 34.94it/s]
Loading 0:  74%|███████▍  | 376/507 [00:25<00:03, 33.83it/s]
Loading 0:  75%|███████▌  | 381/507 [00:25<00:03, 36.40it/s]
Loading 0:  76%|███████▌  | 385/507 [00:25<00:03, 34.60it/s]
Loading 0:  77%|███████▋  | 389/507 [00:26<00:03, 33.77it/s]
Loading 0:  78%|███████▊  | 393/507 [00:26<00:03, 33.28it/s]
Loading 0:  78%|███████▊  | 397/507 [00:26<00:04, 25.23it/s]
Loading 0:  79%|███████▉  | 401/507 [00:26<00:04, 25.12it/s]
Loading 0:  80%|████████  | 408/507 [00:26<00:03, 32.59it/s]
Loading 0:  81%|████████▏ | 412/507 [00:26<00:02, 31.72it/s]
Loading 0:  82%|████████▏ | 416/507 [00:26<00:02, 33.53it/s]
Loading 0:  83%|████████▎ | 420/507 [00:27<00:02, 30.63it/s]
Loading 0:  84%|████████▍ | 426/507 [00:27<00:02, 35.28it/s]
Loading 0:  85%|████████▍ | 430/507 [00:27<00:02, 34.09it/s]
Loading 0:  86%|████████▌ | 435/507 [00:27<00:01, 36.57it/s]
Loading 0:  87%|████████▋ | 439/507 [00:27<00:01, 35.90it/s]
Loading 0:  88%|████████▊ | 444/507 [00:27<00:01, 38.04it/s]
Loading 0:  88%|████████▊ | 448/507 [00:27<00:01, 36.80it/s]
Loading 0:  90%|████████▉ | 454/507 [00:27<00:01, 37.89it/s]
Loading 0:  90%|█████████ | 458/507 [00:30<00:08,  6.02it/s]
Loading 0:  91%|█████████ | 461/507 [00:30<00:06,  7.22it/s]
Loading 0:  92%|█████████▏| 465/507 [00:30<00:04,  9.24it/s]
Loading 0:  93%|█████████▎| 472/507 [00:30<00:02, 14.39it/s]
Loading 0:  94%|█████████▍| 476/507 [00:30<00:01, 16.89it/s]
Loading 0:  95%|█████████▍| 481/507 [00:30<00:01, 20.76it/s]
Loading 0:  96%|█████████▌| 485/507 [00:31<00:00, 22.60it/s]
Loading 0:  96%|█████████▋| 489/507 [00:31<00:00, 25.65it/s]
Loading 0:  97%|█████████▋| 493/507 [00:31<00:00, 26.00it/s]
Loading 0:  98%|█████████▊| 499/507 [00:31<00:00, 31.91it/s]
Loading 0:  99%|█████████▉| 503/507 [00:31<00:00, 32.38it/s]
                                                            
Job mistralai-mistral-small-5341-v91-mkmlizer completed after 166.36s with status: succeeded
Stopping job with name mistralai-mistral-small-5341-v91-mkmlizer
Pipeline stage MKMLizer completed in 167.01s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.23s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-small-5341-v91
Waiting for inference service mistralai-mistral-small-5341-v91 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service mistralai-mistral-small-5341-v91 ready after 221.09136581420898s
Pipeline stage MKMLDeployer completed in 222.39s
run pipeline stage %s
Running pipeline stage StressChecker
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.54197359085083s
Received healthy response to inference request in 3.0420215129852295s
Received healthy response to inference request in 2.4722187519073486s
Received healthy response to inference request in 2.4705216884613037s
Received healthy response to inference request in 2.0936191082000732s
5 requests
0 failed requests
5th percentile: 2.1689996242523195
10th percentile: 2.2443801403045653
20th percentile: 2.3951411724090574
30th percentile: 2.4708611011505126
40th percentile: 2.471539926528931
50th percentile: 2.4722187519073486
60th percentile: 2.500120687484741
70th percentile: 2.528022623062134
80th percentile: 2.64198317527771
90th percentile: 2.84200234413147
95th percentile: 2.9420119285583497
99th percentile: 3.0220195960998537
mean time: 2.524070930480957
Pipeline stage StressChecker completed in 14.96s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.37s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.34s
Shutdown handler de-registered
mistralai-mistral-small_5341_v91 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service mistralai-mistral-small-5341-v91-profiler
Waiting for inference service mistralai-mistral-small-5341-v91-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3264.99s
Shutdown handler de-registered
mistralai-mistral-small_5341_v91 status is now inactive due to auto deactivation removed underperforming models