submission_id: riverise-mistral-0920_v2
developer_uid: Riverise
best_of: 8
celo_rating: 1251.0
display_name: riverise-mistral-0920_v2
family_friendly_score: 0.0
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.15, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 30, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
gpu_counts: {'NVIDIA RTX A5000': 1}
is_internal_developer: False
language_model: Riverise/mistral_0920
latencies: [{'batch_size': 1, 'throughput': 0.6215837596014069, 'latency_mean': 1.608700442314148, 'latency_p50': 1.5998952388763428, 'latency_p90': 1.7747720718383788}, {'batch_size': 3, 'throughput': 1.0927661959911366, 'latency_mean': 2.738461331129074, 'latency_p50': 2.747393250465393, 'latency_p90': 3.0207642793655394}, {'batch_size': 5, 'throughput': 1.2463433068694456, 'latency_mean': 3.9995303654670717, 'latency_p50': 3.9950989484786987, 'latency_p90': 4.460922622680664}, {'batch_size': 6, 'throughput': 1.2813630554247817, 'latency_mean': 4.658743450641632, 'latency_p50': 4.656467080116272, 'latency_p90': 5.304056692123413}, {'batch_size': 8, 'throughput': 1.2631946842837891, 'latency_mean': 6.2944500184059144, 'latency_p50': 6.325100541114807, 'latency_p90': 7.083149027824402}, {'batch_size': 10, 'throughput': 1.2299270726195208, 'latency_mean': 8.07107695221901, 'latency_p50': 8.072474837303162, 'latency_p90': 9.267356157302856}]
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: Riverise/mistral_0920
model_name: riverise-mistral-0920_v2
model_num_parameters: 12772070400.0
model_repo: Riverise/mistral_0920
model_size: 13B
num_battles: 369426
num_wins: 186224
ranking_group: single
status: torndown
submission_type: basic
throughput_3p7s: 1.22
timestamp: 2024-09-21T09:37:41+00:00
us_pacific_date: 2024-09-21
win_ratio: 0.5040901290109521
Download Preference Data
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name riverise-mistral-0920-v2-mkmlizer
Waiting for job on riverise-mistral-0920-v2-mkmlizer to finish
riverise-mistral-0920-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
riverise-mistral-0920-v2-mkmlizer: ║ _____ __ __ ║
riverise-mistral-0920-v2-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
riverise-mistral-0920-v2-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
riverise-mistral-0920-v2-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
riverise-mistral-0920-v2-mkmlizer: ║ /___/ ║
riverise-mistral-0920-v2-mkmlizer: ║ ║
riverise-mistral-0920-v2-mkmlizer: ║ Version: 0.10.1 ║
riverise-mistral-0920-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
riverise-mistral-0920-v2-mkmlizer: ║ https://mk1.ai ║
riverise-mistral-0920-v2-mkmlizer: ║ ║
riverise-mistral-0920-v2-mkmlizer: ║ The license key for the current software has been verified as ║
riverise-mistral-0920-v2-mkmlizer: ║ belonging to: ║
riverise-mistral-0920-v2-mkmlizer: ║ ║
riverise-mistral-0920-v2-mkmlizer: ║ Chai Research Corp. ║
riverise-mistral-0920-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
riverise-mistral-0920-v2-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
riverise-mistral-0920-v2-mkmlizer: ║ ║
riverise-mistral-0920-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
riverise-mistral-0920-v2-mkmlizer: Downloaded to shared memory in 55.410s
riverise-mistral-0920-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpqricghqg, device:0
riverise-mistral-0920-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission mistralai-mistral-small_5341_v11: ('http://mistralai-mistral-small-5341-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-small_5341_v11: ('http://mistralai-mistral-small-5341-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-small_5341_v11: ('http://mistralai-mistral-small-5341-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
riverise-mistral-0920-v2-mkmlizer: quantized model in 35.823s
riverise-mistral-0920-v2-mkmlizer: Processed model Riverise/mistral_0920 in 91.234s
riverise-mistral-0920-v2-mkmlizer: creating bucket guanaco-mkml-models
riverise-mistral-0920-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
riverise-mistral-0920-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/riverise-mistral-0920-v2
riverise-mistral-0920-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/riverise-mistral-0920-v2/config.json
riverise-mistral-0920-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/riverise-mistral-0920-v2/special_tokens_map.json
riverise-mistral-0920-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/riverise-mistral-0920-v2/tokenizer_config.json
riverise-mistral-0920-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/riverise-mistral-0920-v2/tokenizer.json
Failed to get response for submission mistralai-mistral-small_5341_v11: ('http://mistralai-mistral-small-5341-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
riverise-mistral-0920-v2-mkmlizer: Loading 0: 0%| | 0/363 [00:00<?, ?it/s] Loading 0: 2%|▏ | 7/363 [00:00<00:06, 53.55it/s] Loading 0: 5%|▍ | 17/363 [00:00<00:04, 75.86it/s] Loading 0: 9%|▊ | 31/363 [00:00<00:03, 87.46it/s] Loading 0: 11%|█ | 40/363 [00:00<00:03, 86.62it/s] Loading 0: 14%|█▍ | 51/363 [00:00<00:03, 93.88it/s] Loading 0: 17%|█▋ | 61/363 [00:01<00:13, 21.68it/s] Loading 0: 19%|█▉ | 70/363 [00:01<00:10, 27.91it/s] Loading 0: 22%|██▏ | 80/363 [00:01<00:07, 35.97it/s] Loading 0: 24%|██▍ | 88/363 [00:02<00:06, 42.15it/s] Loading 0: 28%|██▊ | 102/363 [00:02<00:04, 58.57it/s] Loading 0: 31%|███ | 112/363 [00:02<00:04, 60.67it/s] Loading 0: 33%|███▎ | 121/363 [00:02<00:03, 65.93it/s] Loading 0: 36%|███▌ | 130/363 [00:02<00:03, 69.68it/s] Loading 0: 39%|███▉ | 141/363 [00:02<00:02, 78.89it/s] Loading 0: 42%|████▏ | 151/363 [00:03<00:09, 23.25it/s] Loading 0: 44%|████▍ | 161/363 [00:03<00:06, 30.11it/s] Loading 0: 48%|████▊ | 175/363 [00:04<00:04, 40.23it/s] Loading 0: 51%|█████ | 184/363 [00:04<00:03, 46.45it/s] Loading 0: 53%|█████▎ | 193/363 [00:04<00:03, 52.54it/s] Loading 0: 56%|█████▌ | 202/363 [00:04<00:02, 59.37it/s] Loading 0: 58%|█████▊ | 212/363 [00:04<00:02, 67.52it/s] Loading 0: 61%|██████ | 221/363 [00:04<00:01, 71.56it/s] Loading 0: 63%|██████▎ | 230/363 [00:05<00:06, 21.87it/s] Loading 0: 66%|██████▌ | 238/363 [00:05<00:04, 26.80it/s] Loading 0: 68%|██████▊ | 247/363 [00:05<00:03, 33.36it/s] Loading 0: 71%|███████ | 256/363 [00:06<00:02, 41.04it/s] Loading 0: 73%|███████▎ | 265/363 [00:06<00:02, 48.91it/s] Loading 0: 75%|███████▌ | 274/363 [00:06<00:01, 54.65it/s] Loading 0: 78%|███████▊ | 283/363 [00:06<00:01, 59.79it/s] Loading 0: 80%|████████ | 292/363 [00:06<00:01, 65.55it/s] Loading 0: 83%|████████▎ | 301/363 [00:06<00:00, 70.80it/s] Loading 0: 85%|████████▌ | 310/363 [00:07<00:02, 20.55it/s] Loading 0: 88%|████████▊ | 319/363 [00:07<00:01, 26.40it/s] Loading 0: 91%|█████████ | 330/363 [00:07<00:00, 35.66it/s] Loading 0: 93%|█████████▎| 338/363 [00:08<00:00, 41.09it/s] Loading 0: 95%|█████████▌| 346/363 [00:08<00:00, 47.26it/s] Loading 0: 98%|█████████▊| 356/363 [00:08<00:00, 57.07it/s]
Job riverise-mistral-0920-v2-mkmlizer completed after 116.12s with status: succeeded
Stopping job with name riverise-mistral-0920-v2-mkmlizer
Pipeline stage MKMLizer completed in 117.06s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service riverise-mistral-0920-v2
Failed to get response for submission mistralai-mistral-small-_5341_v6: ('http://mistralai-mistral-small-5341-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Waiting for inference service riverise-mistral-0920-v2 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission mistralai-mistral-small_5341_v19: ('http://mistralai-mistral-small-5341-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:34868->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Retrying (%r) after connection broken by '%r': %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission blend_pudib_2024-09-20: ('http://mistralai-mistral-small-5341-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:47942->127.0.0.1:8080: read: connection reset by peer\n')
Inference service riverise-mistral-0920-v2 ready after 190.7124683856964s
Pipeline stage MKMLDeployer completed in 197.20s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.280996322631836s
Received healthy response to inference request in 6.679152011871338s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.0545833110809326s
Received healthy response to inference request in 1.799912929534912s
Received healthy response to inference request in 3.0892655849456787s
5 requests
0 failed requests
5th percentile: 1.8508470058441162
10th percentile: 1.9017810821533203
20th percentile: 2.0036492347717285
30th percentile: 2.0998659133911133
40th percentile: 2.1904311180114746
50th percentile: 2.280996322631836
60th percentile: 2.6043040275573732
70th percentile: 2.92761173248291
80th percentile: 3.8072428703308114
90th percentile: 5.243197441101074
95th percentile: 5.961174726486205
99th percentile: 6.5355565547943115
mean time: 3.1807820320129396
Pipeline stage StressChecker completed in 27.83s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 8.84s
Shutdown handler de-registered
riverise-mistral-0920_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service riverise-mistral-0920-v2-profiler
Waiting for inference service riverise-mistral-0920-v2-profiler to be ready
Inference service riverise-mistral-0920-v2-profiler ready after 190.4464247226715s
Pipeline stage MKMLProfilerDeployer completed in 190.84s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/riverise-mistral-0920-v2-profiler-predictor-00001-deploymeljgcw:/code/chaiverse_profiler_1726912057 --namespace tenant-chaiml-guanaco
kubectl exec -it riverise-mistral-0920-v2-profiler-predictor-00001-deploymeljgcw --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726912057 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1726912057/summary.json'
kubectl exec -it riverise-mistral-0920-v2-profiler-predictor-00001-deploymeljgcw --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726912057/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1157.66s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service riverise-mistral-0920-v2-profiler is running
Tearing down inference service riverise-mistral-0920-v2-profiler
Service riverise-mistral-0920-v2-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 11.20s
Shutdown handler de-registered
riverise-mistral-0920_v2 status is now inactive due to auto deactivation removed underperforming models
riverise-mistral-0920_v2 status is now torndown due to DeploymentManager action