riverise-mistral-0920

submission_id: riverise-mistral-0920_v2

developer_uid: Riverise

best_of: 8

celo_rating: 1251.0

display_name: riverise-mistral-0920_v2

family_friendly_score: 0.0

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 1.15, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 30, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

gpu_counts: {'NVIDIA RTX A5000': 1}

is_internal_developer: False

language_model: Riverise/mistral_0920

latencies: [{'batch_size': 1, 'throughput': 0.6215837596014069, 'latency_mean': 1.608700442314148, 'latency_p50': 1.5998952388763428, 'latency_p90': 1.7747720718383788}, {'batch_size': 3, 'throughput': 1.0927661959911366, 'latency_mean': 2.738461331129074, 'latency_p50': 2.747393250465393, 'latency_p90': 3.0207642793655394}, {'batch_size': 5, 'throughput': 1.2463433068694456, 'latency_mean': 3.9995303654670717, 'latency_p50': 3.9950989484786987, 'latency_p90': 4.460922622680664}, {'batch_size': 6, 'throughput': 1.2813630554247817, 'latency_mean': 4.658743450641632, 'latency_p50': 4.656467080116272, 'latency_p90': 5.304056692123413}, {'batch_size': 8, 'throughput': 1.2631946842837891, 'latency_mean': 6.2944500184059144, 'latency_p50': 6.325100541114807, 'latency_p90': 7.083149027824402}, {'batch_size': 10, 'throughput': 1.2299270726195208, 'latency_mean': 8.07107695221901, 'latency_p50': 8.072474837303162, 'latency_p90': 9.267356157302856}]

max_input_tokens: 1024

max_output_tokens: 64

model_architecture: MistralForCausalLM

model_group: Riverise/mistral_0920

model_name: riverise-mistral-0920_v2

model_num_parameters: 12772070400.0

model_repo: Riverise/mistral_0920

model_size: 13B

num_battles: 369426

num_wins: 186224

ranking_group: single

status: torndown

submission_type: basic

throughput_3p7s: 1.22

timestamp: 2024-09-21T09:37:41+00:00

us_pacific_date: 2024-09-21

win_ratio: 0.5040901290109521

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name riverise-mistral-0920-v2-mkmlizer
Waiting for job on riverise-mistral-0920-v2-mkmlizer to finish
riverise-mistral-0920-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
riverise-mistral-0920-v2-mkmlizer: ║     _____            __           __                                ║
riverise-mistral-0920-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
riverise-mistral-0920-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
riverise-mistral-0920-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
riverise-mistral-0920-v2-mkmlizer: ║       /___/                                                         ║
riverise-mistral-0920-v2-mkmlizer: ║                                                                     ║
riverise-mistral-0920-v2-mkmlizer: ║  Version: 0.10.1                                                    ║
riverise-mistral-0920-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
riverise-mistral-0920-v2-mkmlizer: ║  https://mk1.ai                                                     ║
riverise-mistral-0920-v2-mkmlizer: ║                                                                     ║
riverise-mistral-0920-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
riverise-mistral-0920-v2-mkmlizer: ║  belonging to:                                                      ║
riverise-mistral-0920-v2-mkmlizer: ║                                                                     ║
riverise-mistral-0920-v2-mkmlizer: ║  Chai Research Corp.                                                ║
riverise-mistral-0920-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
riverise-mistral-0920-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
riverise-mistral-0920-v2-mkmlizer: ║                                                                     ║
riverise-mistral-0920-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
riverise-mistral-0920-v2-mkmlizer: Downloaded to shared memory in 55.410s
riverise-mistral-0920-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpqricghqg, device:0
riverise-mistral-0920-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission mistralai-mistral-small_5341_v11: ('http://mistralai-mistral-small-5341-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-small_5341_v11: ('http://mistralai-mistral-small-5341-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-small_5341_v11: ('http://mistralai-mistral-small-5341-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
riverise-mistral-0920-v2-mkmlizer: quantized model in 35.823s
riverise-mistral-0920-v2-mkmlizer: Processed model Riverise/mistral_0920 in 91.234s
riverise-mistral-0920-v2-mkmlizer: creating bucket guanaco-mkml-models
riverise-mistral-0920-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
riverise-mistral-0920-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/riverise-mistral-0920-v2
riverise-mistral-0920-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/riverise-mistral-0920-v2/config.json
riverise-mistral-0920-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/riverise-mistral-0920-v2/special_tokens_map.json
riverise-mistral-0920-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/riverise-mistral-0920-v2/tokenizer_config.json
riverise-mistral-0920-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/riverise-mistral-0920-v2/tokenizer.json
Failed to get response for submission mistralai-mistral-small_5341_v11: ('http://mistralai-mistral-small-5341-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
riverise-mistral-0920-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 7/363 [00:00<00:06, 53.55it/s]
Loading 0:   5%|▍         | 17/363 [00:00<00:04, 75.86it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:03, 87.46it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:03, 86.62it/s]
Loading 0:  14%|█▍        | 51/363 [00:00<00:03, 93.88it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:13, 21.68it/s]
Loading 0:  19%|█▉        | 70/363 [00:01<00:10, 27.91it/s]
Loading 0:  22%|██▏       | 80/363 [00:01<00:07, 35.97it/s]
Loading 0:  24%|██▍       | 88/363 [00:02<00:06, 42.15it/s]
Loading 0:  28%|██▊       | 102/363 [00:02<00:04, 58.57it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:04, 60.67it/s]
Loading 0:  33%|███▎      | 121/363 [00:02<00:03, 65.93it/s]
Loading 0:  36%|███▌      | 130/363 [00:02<00:03, 69.68it/s]
Loading 0:  39%|███▉      | 141/363 [00:02<00:02, 78.89it/s]
Loading 0:  42%|████▏     | 151/363 [00:03<00:09, 23.25it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:06, 30.11it/s]
Loading 0:  48%|████▊     | 175/363 [00:04<00:04, 40.23it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:03, 46.45it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 52.54it/s]
Loading 0:  56%|█████▌    | 202/363 [00:04<00:02, 59.37it/s]
Loading 0:  58%|█████▊    | 212/363 [00:04<00:02, 67.52it/s]
Loading 0:  61%|██████    | 221/363 [00:04<00:01, 71.56it/s]
Loading 0:  63%|██████▎   | 230/363 [00:05<00:06, 21.87it/s]
Loading 0:  66%|██████▌   | 238/363 [00:05<00:04, 26.80it/s]
Loading 0:  68%|██████▊   | 247/363 [00:05<00:03, 33.36it/s]
Loading 0:  71%|███████   | 256/363 [00:06<00:02, 41.04it/s]
Loading 0:  73%|███████▎  | 265/363 [00:06<00:02, 48.91it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:01, 54.65it/s]
Loading 0:  78%|███████▊  | 283/363 [00:06<00:01, 59.79it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 65.55it/s]
Loading 0:  83%|████████▎ | 301/363 [00:06<00:00, 70.80it/s]
Loading 0:  85%|████████▌ | 310/363 [00:07<00:02, 20.55it/s]
Loading 0:  88%|████████▊ | 319/363 [00:07<00:01, 26.40it/s]
Loading 0:  91%|█████████ | 330/363 [00:07<00:00, 35.66it/s]
Loading 0:  93%|█████████▎| 338/363 [00:08<00:00, 41.09it/s]
Loading 0:  95%|█████████▌| 346/363 [00:08<00:00, 47.26it/s]
Loading 0:  98%|█████████▊| 356/363 [00:08<00:00, 57.07it/s]
                                                            
Job riverise-mistral-0920-v2-mkmlizer completed after 116.12s with status: succeeded
Stopping job with name riverise-mistral-0920-v2-mkmlizer
Pipeline stage MKMLizer completed in 117.06s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service riverise-mistral-0920-v2
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Waiting for inference service riverise-mistral-0920-v2 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission mistralai-mistral-small_5341_v19: ('http://mistralai-mistral-small-5341-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:34868->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Retrying (%r) after connection broken by '%r': %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission blend_pudib_2024-09-20: ('http://mistralai-mistral-small-5341-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:47942->127.0.0.1:8080: read: connection reset by peer\n')
Inference service riverise-mistral-0920-v2 ready after 190.7124683856964s
Pipeline stage MKMLDeployer completed in 197.20s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.280996322631836s
Received healthy response to inference request in 6.679152011871338s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.0545833110809326s
Received healthy response to inference request in 1.799912929534912s
Received healthy response to inference request in 3.0892655849456787s
5 requests
0 failed requests
5th percentile: 1.8508470058441162
10th percentile: 1.9017810821533203
20th percentile: 2.0036492347717285
30th percentile: 2.0998659133911133
40th percentile: 2.1904311180114746
50th percentile: 2.280996322631836
60th percentile: 2.6043040275573732
70th percentile: 2.92761173248291
80th percentile: 3.8072428703308114
90th percentile: 5.243197441101074
95th percentile: 5.961174726486205
99th percentile: 6.5355565547943115
mean time: 3.1807820320129396
Pipeline stage StressChecker completed in 27.83s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 8.84s
Shutdown handler de-registered
riverise-mistral-0920_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service riverise-mistral-0920-v2-profiler
Waiting for inference service riverise-mistral-0920-v2-profiler to be ready
Inference service riverise-mistral-0920-v2-profiler ready after 190.4464247226715s
Pipeline stage MKMLProfilerDeployer completed in 190.84s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/riverise-mistral-0920-v2-profiler-predictor-00001-deploymeljgcw:/code/chaiverse_profiler_1726912057 --namespace tenant-chaiml-guanaco
kubectl exec -it riverise-mistral-0920-v2-profiler-predictor-00001-deploymeljgcw --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726912057 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1726912057/summary.json'
kubectl exec -it riverise-mistral-0920-v2-profiler-predictor-00001-deploymeljgcw --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726912057/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1157.66s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service riverise-mistral-0920-v2-profiler is running
Tearing down inference service riverise-mistral-0920-v2-profiler
Service riverise-mistral-0920-v2-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 11.20s
Shutdown handler de-registered
riverise-mistral-0920_v2 status is now inactive due to auto deactivation removed underperforming models
riverise-mistral-0920_v2 status is now torndown due to DeploymentManager action