mistralai-mistral-small_5341

developer_uid: robert_irvine

submission_id: mistralai-mistral-small_5341_v27

model_name: mistralai-mistral-small_5341_v27

model_group: mistralai/Mistral-Small-

status: torndown

timestamp: 2024-09-23T20:03:47+00:00

num_battles: 12319

num_wins: 6249

celo_rating: 1242.48

family_friendly_score: 0.0

submission_type: basic

model_repo: mistralai/Mistral-Small-Instruct-2409

model_architecture: MistralForCausalLM

model_num_parameters: 22247282688.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.37949202016412725, 'latency_mean': 2.6350243878364563, 'latency_p50': 2.6284338235855103, 'latency_p90': 2.911684036254883}, {'batch_size': 2, 'throughput': 0.5930837152412876, 'latency_mean': 3.364334923028946, 'latency_p50': 3.3436378240585327, 'latency_p90': 3.695647954940796}, {'batch_size': 3, 'throughput': 0.7477580894799926, 'latency_mean': 3.9997671914100645, 'latency_p50': 4.045446395874023, 'latency_p90': 4.363637518882752}, {'batch_size': 4, 'throughput': 0.8634132126960634, 'latency_mean': 4.611439305543899, 'latency_p50': 4.635320067405701, 'latency_p90': 5.121024632453919}, {'batch_size': 5, 'throughput': 0.9502973729769795, 'latency_mean': 5.229388775825501, 'latency_p50': 5.219917178153992, 'latency_p90': 5.836426424980163}]

gpu_counts: {'NVIDIA RTX A6000': 1}

display_name: mistralai-mistral-small_5341_v27

is_internal_developer: True

language_model: mistralai/Mistral-Small-Instruct-2409

model_size: 22B

ranking_group: single

throughput_3p7s: 0.68

us_pacific_date: 2024-09-23

win_ratio: 0.5072652000974105

generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '###', 'Bot:', 'User:', 'You:', '<|im_end|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-small-5341-v27-mkmlizer
Waiting for job on mistralai-mistral-small-5341-v27-mkmlizer to finish
mistralai-mistral-small-5341-v27-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-small-5341-v27-mkmlizer: ║     _____            __           __                                ║
mistralai-mistral-small-5341-v27-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
mistralai-mistral-small-5341-v27-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
mistralai-mistral-small-5341-v27-mkmlizer: ║       /___/                                                         ║
mistralai-mistral-small-5341-v27-mkmlizer: ║                                                                     ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  Version: 0.10.1                                                    ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mistral-small-5341-v27-mkmlizer: ║                                                                     ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  belonging to:                                                      ║
mistralai-mistral-small-5341-v27-mkmlizer: ║                                                                     ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mistral-small-5341-v27-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
mistralai-mistral-small-5341-v27-mkmlizer: ║                                                                     ║
mistralai-mistral-small-5341-v27-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
mistralai-mistral-small-5341-v27-mkmlizer: Downloaded to shared memory in 81.014s
mistralai-mistral-small-5341-v27-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp2_f024q8, device:0
mistralai-mistral-small-5341-v27-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
mistralai-mistral-small-5341-v27-mkmlizer: quantized model in 44.363s
mistralai-mistral-small-5341-v27-mkmlizer: Processed model mistralai/Mistral-Small-Instruct-2409 in 125.377s
mistralai-mistral-small-5341-v27-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-small-5341-v27-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-small-5341-v27-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-small-5341-v27
mistralai-mistral-small-5341-v27-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-small-5341-v27/config.json
mistralai-mistral-small-5341-v27-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-small-5341-v27/special_tokens_map.json
mistralai-mistral-small-5341-v27-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-small-5341-v27/tokenizer_config.json
mistralai-mistral-small-5341-v27-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/mistralai-mistral-small-5341-v27/tokenizer.model
mistralai-mistral-small-5341-v27-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-small-5341-v27/tokenizer.json
mistralai-mistral-small-5341-v27-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/mistralai-mistral-small-5341-v27/flywheel_model.1.safetensors
mistralai-mistral-small-5341-v27-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-small-5341-v27/flywheel_model.0.safetensors
mistralai-mistral-small-5341-v27-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:18, 27.33it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:12, 40.01it/s]
Loading 0:   3%|▎         | 17/507 [00:00<00:13, 36.39it/s]
Loading 0:   4%|▍         | 21/507 [00:00<00:13, 36.64it/s]
Loading 0:   5%|▍         | 25/507 [00:00<00:13, 34.58it/s]
Loading 0:   6%|▌         | 30/507 [00:00<00:13, 36.31it/s]
Loading 0:   7%|▋         | 34/507 [00:00<00:13, 34.60it/s]
Loading 0:   8%|▊         | 39/507 [00:01<00:12, 36.33it/s]
Loading 0:   8%|▊         | 43/507 [00:01<00:13, 34.36it/s]
Loading 0:   9%|▉         | 47/507 [00:01<00:13, 33.30it/s]
Loading 0:  10%|█         | 51/507 [00:01<00:14, 32.15it/s]
Loading 0:  11%|█         | 55/507 [00:01<00:20, 22.21it/s]
Loading 0:  11%|█▏        | 58/507 [00:01<00:19, 23.19it/s]
Loading 0:  12%|█▏        | 63/507 [00:02<00:16, 27.09it/s]
Loading 0:  13%|█▎        | 67/507 [00:02<00:15, 27.95it/s]
Loading 0:  14%|█▍        | 72/507 [00:02<00:13, 32.00it/s]
Loading 0:  15%|█▍        | 76/507 [00:02<00:12, 33.17it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:13, 30.60it/s]
Loading 0:  17%|█▋        | 87/507 [00:02<00:11, 38.18it/s]
Loading 0:  18%|█▊        | 92/507 [00:02<00:10, 38.67it/s]
Loading 0:  19%|█▉        | 97/507 [00:02<00:10, 38.45it/s]
Loading 0:  20%|██        | 102/507 [00:03<00:10, 40.34it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:11, 34.74it/s]
Loading 0:  22%|██▏       | 113/507 [00:03<00:13, 29.27it/s]
Loading 0:  23%|██▎       | 117/507 [00:03<00:13, 28.22it/s]
Loading 0:  24%|██▍       | 122/507 [00:03<00:13, 28.51it/s]
Loading 0:  25%|██▌       | 129/507 [00:03<00:10, 34.70it/s]
Loading 0:  26%|██▌       | 133/507 [00:04<00:11, 33.87it/s]
Loading 0:  27%|██▋       | 138/507 [00:04<00:10, 35.96it/s]
Loading 0:  28%|██▊       | 142/507 [00:04<00:10, 35.26it/s]
Loading 0:  29%|██▉       | 147/507 [00:04<00:09, 37.60it/s]
Loading 0:  30%|██▉       | 151/507 [00:04<00:09, 36.48it/s]
Loading 0:  31%|███       | 156/507 [00:04<00:09, 38.10it/s]
Loading 0:  32%|███▏      | 160/507 [00:04<00:09, 36.24it/s]
Loading 0:  32%|███▏      | 164/507 [00:04<00:10, 34.16it/s]
Loading 0:  33%|███▎      | 169/507 [00:05<00:14, 23.92it/s]
Loading 0:  34%|███▍      | 172/507 [00:05<00:14, 23.49it/s]
Loading 0:  35%|███▍      | 176/507 [00:05<00:13, 24.20it/s]
Loading 0:  36%|███▌      | 181/507 [00:05<00:11, 29.26it/s]
Loading 0:  36%|███▋      | 185/507 [00:05<00:11, 27.82it/s]
Loading 0:  38%|███▊      | 192/507 [00:05<00:09, 34.94it/s]
Loading 0:  39%|███▊      | 196/507 [00:06<00:09, 34.30it/s]
Loading 0:  40%|███▉      | 201/507 [00:06<00:08, 36.67it/s]
Loading 0:  40%|████      | 205/507 [00:06<00:08, 34.38it/s]
Loading 0:  41%|████▏     | 210/507 [00:06<00:08, 35.87it/s]
Loading 0:  42%|████▏     | 214/507 [00:06<00:08, 34.60it/s]
Loading 0:  43%|████▎     | 218/507 [00:06<00:08, 34.82it/s]
Loading 0:  44%|████▍     | 222/507 [00:06<00:08, 35.07it/s]
Loading 0:  45%|████▍     | 226/507 [00:07<00:11, 24.43it/s]
Loading 0:  45%|████▌     | 230/507 [00:07<00:11, 24.93it/s]
Loading 0:  47%|████▋     | 237/507 [00:07<00:08, 32.69it/s]
Loading 0:  48%|████▊     | 241/507 [00:07<00:08, 32.45it/s]
Loading 0:  49%|████▊     | 246/507 [00:07<00:07, 34.41it/s]
Loading 0:  49%|████▉     | 250/507 [00:07<00:07, 33.32it/s]
Loading 0:  50%|█████     | 255/507 [00:07<00:07, 35.68it/s]
Loading 0:  51%|█████     | 259/507 [00:07<00:07, 34.56it/s]
Loading 0:  52%|█████▏    | 264/507 [00:08<00:06, 35.88it/s]
Loading 0:  53%|█████▎    | 268/507 [00:08<00:06, 35.35it/s]
Loading 0:  54%|█████▍    | 273/507 [00:08<00:06, 37.65it/s]
Loading 0:  55%|█████▍    | 277/507 [00:08<00:06, 36.60it/s]
Loading 0:  56%|█████▌    | 283/507 [00:08<00:05, 38.17it/s]
Loading 0:  57%|█████▋    | 287/507 [00:08<00:09, 23.05it/s]
Loading 0:  58%|█████▊    | 293/507 [00:09<00:08, 26.69it/s]
Loading 0:  59%|█████▉    | 299/507 [00:23<00:07, 26.69it/s]
Loading 0:  59%|█████▉    | 300/507 [00:23<02:46,  1.25it/s]
Loading 0:  60%|█████▉    | 302/507 [00:23<02:24,  1.42it/s]
Loading 0:  61%|██████    | 307/507 [00:24<01:37,  2.05it/s]
Loading 0:  61%|██████    | 310/507 [00:24<01:17,  2.53it/s]
Loading 0:  62%|██████▏   | 314/507 [00:24<00:55,  3.47it/s]
Loading 0:  63%|██████▎   | 319/507 [00:24<00:37,  5.04it/s]
Loading 0:  64%|██████▍   | 324/507 [00:24<00:25,  7.11it/s]
Loading 0:  65%|██████▍   | 328/507 [00:24<00:19,  9.08it/s]
Loading 0:  66%|██████▌   | 333/507 [00:24<00:14, 12.34it/s]
Loading 0:  66%|██████▋   | 337/507 [00:24<00:11, 14.85it/s]
Loading 0:  67%|██████▋   | 341/507 [00:25<00:11, 14.49it/s]
Loading 0:  68%|██████▊   | 345/507 [00:25<00:09, 16.78it/s]
Loading 0:  69%|██████▊   | 348/507 [00:25<00:08, 18.41it/s]
Loading 0:  70%|██████▉   | 354/507 [00:25<00:06, 24.57it/s]
Loading 0:  71%|███████   | 358/507 [00:25<00:05, 26.68it/s]
Loading 0:  72%|███████▏  | 363/507 [00:25<00:04, 30.27it/s]
Loading 0:  72%|███████▏  | 367/507 [00:25<00:04, 30.28it/s]
Loading 0:  73%|███████▎  | 371/507 [00:26<00:04, 32.19it/s]
Loading 0:  74%|███████▍  | 375/507 [00:26<00:04, 28.98it/s]
Loading 0:  75%|███████▍  | 379/507 [00:26<00:04, 31.24it/s]
Loading 0:  76%|███████▌  | 383/507 [00:26<00:04, 28.89it/s]
Loading 0:  77%|███████▋  | 389/507 [00:26<00:03, 34.62it/s]
Loading 0:  78%|███████▊  | 393/507 [00:26<00:03, 34.12it/s]
Loading 0:  78%|███████▊  | 397/507 [00:26<00:04, 26.17it/s]
Loading 0:  79%|███████▉  | 401/507 [00:27<00:04, 26.37it/s]
Loading 0:  80%|████████  | 408/507 [00:27<00:02, 33.20it/s]
Loading 0:  81%|████████▏ | 412/507 [00:27<00:02, 32.80it/s]
Loading 0:  82%|████████▏ | 417/507 [00:27<00:02, 35.07it/s]
Loading 0:  83%|████████▎ | 421/507 [00:27<00:02, 34.86it/s]
Loading 0:  84%|████████▍ | 426/507 [00:27<00:02, 37.01it/s]
Loading 0:  85%|████████▍ | 430/507 [00:27<00:02, 37.02it/s]
Loading 0:  86%|████████▌ | 435/507 [00:27<00:01, 39.44it/s]
Loading 0:  87%|████████▋ | 440/507 [00:28<00:01, 39.82it/s]
Loading 0:  88%|████████▊ | 445/507 [00:28<00:01, 39.97it/s]
Loading 0:  89%|████████▉ | 450/507 [00:28<00:01, 42.18it/s]
Loading 0:  90%|████████▉ | 455/507 [00:30<00:08,  6.30it/s]
Loading 0:  91%|█████████ | 459/507 [00:30<00:06,  7.83it/s]
Loading 0:  92%|█████████▏| 465/507 [00:30<00:03, 10.93it/s]
Loading 0:  93%|█████████▎| 472/507 [00:31<00:02, 15.78it/s]
Loading 0:  94%|█████████▍| 477/507 [00:31<00:01, 18.98it/s]
Loading 0:  95%|█████████▌| 482/507 [00:31<00:01, 22.41it/s]
Loading 0:  96%|█████████▌| 487/507 [00:31<00:00, 26.49it/s]
Loading 0:  97%|█████████▋| 492/507 [00:31<00:00, 26.93it/s]
Loading 0:  98%|█████████▊| 499/507 [00:31<00:00, 33.79it/s]
Loading 0:  99%|█████████▉| 504/507 [00:31<00:00, 34.39it/s]
                                                            
Job mistralai-mistral-small-5341-v27-mkmlizer completed after 150.15s with status: succeeded
Stopping job with name mistralai-mistral-small-5341-v27-mkmlizer
Pipeline stage MKMLizer completed in 152.30s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.08s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-small-5341-v27
Waiting for inference service mistralai-mistral-small-5341-v27 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service mistralai-mistral-small-5341-v27 ready after 201.19302105903625s
Pipeline stage MKMLDeployer completed in 201.60s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.8586111068725586s
Received healthy response to inference request in 2.545760154724121s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.1555168628692627s
Received healthy response to inference request in 1.3610379695892334s
Received healthy response to inference request in 1.6162819862365723s
5 requests
0 failed requests
5th percentile: 1.4120867729187012
10th percentile: 1.463135576248169
20th percentile: 1.5652331829071044
30th percentile: 1.7241289615631104
40th percentile: 1.9398229122161865
50th percentile: 2.1555168628692627
60th percentile: 2.311614179611206
70th percentile: 2.4677114963531492
80th percentile: 2.6083303451538087
90th percentile: 2.7334707260131834
95th percentile: 2.796040916442871
99th percentile: 2.846097068786621
mean time: 2.10744161605835
Pipeline stage StressChecker completed in 11.92s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 6.03s
Shutdown handler de-registered
mistralai-mistral-small_5341_v27 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service mistralai-mistral-small-5341-v27-profiler
Waiting for inference service mistralai-mistral-small-5341-v27-profiler to be ready
Inference service mistralai-mistral-small-5341-v27-profiler ready after 190.43198823928833s
Pipeline stage MKMLProfilerDeployer completed in 190.85s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/mistralai-mistral-sm045a07c44408d5f165a98a5984625fa7-deplohg7k6:/code/chaiverse_profiler_1727122446 --namespace tenant-chaiml-guanaco
kubectl exec -it mistralai-mistral-sm045a07c44408d5f165a98a5984625fa7-deplohg7k6 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1727122446 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1727122446/summary.json'
kubectl exec -it mistralai-mistral-sm045a07c44408d5f165a98a5984625fa7-deplohg7k6 --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1727122446/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1579.40s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service mistralai-mistral-small-5341-v27-profiler is running
Tearing down inference service mistralai-mistral-small-5341-v27-profiler
Service mistralai-mistral-small-5341-v27-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.38s
Shutdown handler de-registered
mistralai-mistral-small_5341_v27 status is now inactive due to auto deactivation removed underperforming models
mistralai-mistral-small_5341_v27 status is now torndown due to DeploymentManager action