submission_id: trace2333-mistral-trial6_v3
developer_uid: Trace2333
alignment_samples: 11891
alignment_score: -0.28571637490810875
best_of: 8
celo_rating: 1259.59
display_name: trace2333-mistral-trial6_v3
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.06, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '###'], 'max_input_tokens': 512, 'best_of': 8, 'max_output_tokens': 64}
gpu_counts: {'NVIDIA RTX A5000': 1}
is_internal_developer: False
language_model: Trace2333/mistral_trial6
latencies: [{'batch_size': 1, 'throughput': 0.6895212327046406, 'latency_mean': 1.4501783382892608, 'latency_p50': 1.4523802995681763, 'latency_p90': 1.6143239259719848}, {'batch_size': 3, 'throughput': 1.3347311799296235, 'latency_mean': 2.2425858199596407, 'latency_p50': 2.245273470878601, 'latency_p90': 2.4768696546554567}, {'batch_size': 5, 'throughput': 1.5669709396077505, 'latency_mean': 3.1724313282966614, 'latency_p50': 3.166548728942871, 'latency_p90': 3.564660167694092}, {'batch_size': 6, 'throughput': 1.6215319129300012, 'latency_mean': 3.6815018010139466, 'latency_p50': 3.669707775115967, 'latency_p90': 4.153834319114685}, {'batch_size': 8, 'throughput': 1.6218899584964699, 'latency_mean': 4.905808510780335, 'latency_p50': 4.942691802978516, 'latency_p90': 5.553716039657592}, {'batch_size': 10, 'throughput': 1.5487342890351088, 'latency_mean': 6.418278940916061, 'latency_p50': 6.4532681703567505, 'latency_p90': 7.308920049667359}]
max_input_tokens: 512
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: Trace2333/mistral_trial6
model_name: trace2333-mistral-trial6_v3
model_num_parameters: 12772070400.0
model_repo: Trace2333/mistral_trial6
model_size: 13B
num_battles: 11891
num_wins: 6354
propriety_score: 0.7383444338725024
propriety_total_count: 1051.0
ranking_group: single
status: inactive
submission_type: basic
throughput_3p7s: 1.63
timestamp: 2024-09-06T15:34:03+00:00
us_pacific_date: 2024-09-06
win_ratio: 0.5343537128921033
Download Preference Data
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name trace2333-mistral-trial6-v3-mkmlizer
Waiting for job on trace2333-mistral-trial6-v3-mkmlizer to finish
trace2333-mistral-trial6-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
trace2333-mistral-trial6-v3-mkmlizer: ║     _____            __           __                                ║
trace2333-mistral-trial6-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
trace2333-mistral-trial6-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
trace2333-mistral-trial6-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
trace2333-mistral-trial6-v3-mkmlizer: ║       /___/                                                         ║
trace2333-mistral-trial6-v3-mkmlizer: ║                                                                     ║
trace2333-mistral-trial6-v3-mkmlizer: ║  Version: 0.10.1                                                    ║
trace2333-mistral-trial6-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
trace2333-mistral-trial6-v3-mkmlizer: ║  https://mk1.ai                                                     ║
trace2333-mistral-trial6-v3-mkmlizer: ║                                                                     ║
trace2333-mistral-trial6-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
trace2333-mistral-trial6-v3-mkmlizer: ║  belonging to:                                                      ║
trace2333-mistral-trial6-v3-mkmlizer: ║                                                                     ║
trace2333-mistral-trial6-v3-mkmlizer: ║  Chai Research Corp.                                                ║
trace2333-mistral-trial6-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
trace2333-mistral-trial6-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
trace2333-mistral-trial6-v3-mkmlizer: ║                                                                     ║
trace2333-mistral-trial6-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
trace2333-mistral-trial6-v3-mkmlizer: Downloaded to shared memory in 31.828s
trace2333-mistral-trial6-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpvuuzqwtj, device:0
trace2333-mistral-trial6-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission zonemercy-base-story-v1_v7: ('http://zonemercy-base-story-v1-v7-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
trace2333-mistral-trial6-v3-mkmlizer: quantized model in 36.507s
trace2333-mistral-trial6-v3-mkmlizer: Processed model Trace2333/mistral_trial6 in 68.335s
trace2333-mistral-trial6-v3-mkmlizer: creating bucket guanaco-mkml-models
trace2333-mistral-trial6-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/trace2333-mistral-trial6-v3/special_tokens_map.json
trace2333-mistral-trial6-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/trace2333-mistral-trial6-v3/tokenizer_config.json
trace2333-mistral-trial6-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/trace2333-mistral-trial6-v3/tokenizer.json
trace2333-mistral-trial6-v3-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/trace2333-mistral-trial6-v3/flywheel_model.0.safetensors
trace2333-mistral-trial6-v3-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 7/363 [00:00<00:08, 44.00it/s]
Loading 0:   6%|▌         | 22/363 [00:00<00:04, 77.88it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:04, 80.36it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:04, 74.81it/s]
Loading 0:  13%|█▎        | 49/363 [00:00<00:04, 74.31it/s]
Loading 0:  16%|█▌        | 58/363 [00:00<00:03, 78.03it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:14, 20.50it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:12, 23.99it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:09, 29.31it/s]
Loading 0:  24%|██▍       | 88/363 [00:02<00:07, 36.82it/s]
Loading 0:  27%|██▋       | 97/363 [00:02<00:06, 42.37it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:05, 46.23it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:04, 52.75it/s]
Loading 0:  34%|███▍      | 124/363 [00:02<00:04, 58.74it/s]
Loading 0:  37%|███▋      | 133/363 [00:02<00:03, 62.86it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:11, 19.93it/s]
Loading 0:  42%|████▏     | 151/363 [00:04<00:08, 25.17it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 31.74it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 37.62it/s]
Loading 0:  49%|████▉     | 178/363 [00:04<00:04, 44.43it/s]
Loading 0:  52%|█████▏    | 187/363 [00:04<00:03, 50.28it/s]
Loading 0:  54%|█████▍    | 196/363 [00:04<00:03, 55.43it/s]
Loading 0:  56%|█████▋    | 205/363 [00:04<00:02, 57.67it/s]
Loading 0:  59%|█████▉    | 214/363 [00:05<00:02, 59.02it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:07, 19.90it/s]
Loading 0:  64%|██████▍   | 232/363 [00:06<00:05, 25.59it/s]
Loading 0:  66%|██████▋   | 241/363 [00:06<00:03, 31.31it/s]
Loading 0:  69%|██████▉   | 250/363 [00:06<00:02, 38.49it/s]
Loading 0:  71%|███████▏  | 259/363 [00:06<00:02, 45.79it/s]
Loading 0:  74%|███████▍  | 268/363 [00:06<00:01, 51.12it/s]
Loading 0:  76%|███████▋  | 277/363 [00:06<00:01, 57.24it/s]
Loading 0:  79%|███████▉  | 286/363 [00:07<00:01, 58.81it/s]
Loading 0:  81%|████████▏ | 295/363 [00:07<00:01, 59.56it/s]
Loading 0:  84%|████████▎ | 304/363 [00:08<00:02, 20.13it/s]
Loading 0:  86%|████████▌ | 313/363 [00:08<00:01, 25.87it/s]
Loading 0:  89%|████████▊ | 322/363 [00:08<00:01, 32.10it/s]
Loading 0:  91%|█████████ | 331/363 [00:08<00:00, 39.13it/s]
Loading 0:  94%|█████████▎| 340/363 [00:08<00:00, 46.49it/s]
Loading 0:  96%|█████████▌| 349/363 [00:08<00:00, 52.72it/s]
Loading 0:  99%|█████████▊| 358/363 [00:09<00:00, 56.02it/s]
                                                            

Job trace2333-mistral-trial6-v3-mkmlizer completed after 97.49s with status: succeeded
Stopping job with name trace2333-mistral-trial6-v3-mkmlizer
Pipeline stage MKMLizer completed in 98.70s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service trace2333-mistral-trial6-v3
Waiting for inference service trace2333-mistral-trial6-v3 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission zonemercy-lexical-nemo-_1518_v27: ('http://zonemercy-lexical-nemo-1518-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
Failed to get response for submission zonemercy-base-story-v1_v8: ('http://zonemercy-base-story-v1-v8-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
Failed to get response for submission zonemercy-lexical-nemo-_1518_v27: ('http://zonemercy-lexical-nemo-1518-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
Inference service trace2333-mistral-trial6-v3 ready after 150.84913873672485s
Pipeline stage MKMLDeployer completed in 151.29s
run pipeline stage %s
Running pipeline stage StressChecker
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.2550323009490967s
Received healthy response to inference request in 1.7232716083526611s
Received healthy response to inference request in 1.6905319690704346s
Received healthy response to inference request in 2.5400590896606445s
Received healthy response to inference request in 2.246304750442505s
5 requests
0 failed requests
5th percentile: 1.6970798969268799
10th percentile: 1.7036278247833252
20th percentile: 1.7167236804962158
30th percentile: 1.8278782367706299
40th percentile: 2.0370914936065674
50th percentile: 2.246304750442505
60th percentile: 2.2497957706451417
70th percentile: 2.2532867908477785
80th percentile: 2.3120376586914064
90th percentile: 2.4260483741760255
95th percentile: 2.483053731918335
99th percentile: 2.5286580181121825
mean time: 2.091039943695068
Pipeline stage StressChecker completed in 11.48s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Failed to get response for submission zonemercy-base-story-v1_v8: ('http://zonemercy-base-story-v1-v8-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.82s
Shutdown handler de-registered
trace2333-mistral-trial6_v3 status is now deployed due to DeploymentManager action
Connection pool is full, discarding connection: %s. Connection pool size: %s
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service trace2333-mistral-trial6-v3-profiler
Waiting for inference service trace2333-mistral-trial6-v3-profiler to be ready
Inference service trace2333-mistral-trial6-v3-profiler ready after 150.34203720092773s
Pipeline stage MKMLProfilerDeployer completed in 150.73s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/trace2333-mistral-trial6-v3-profiler-predictor-00001-deplocmvg4:/code/chaiverse_profiler_1725637299 --namespace tenant-chaiml-guanaco
kubectl exec -it trace2333-mistral-trial6-v3-profiler-predictor-00001-deplocmvg4 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1725637299 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725637299/summary.json'
kubectl exec -it trace2333-mistral-trial6-v3-profiler-predictor-00001-deplocmvg4 --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1725637299/summary.json'
Pipeline stage MKMLProfilerRunner completed in 948.79s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service trace2333-mistral-trial6-v3-profiler is running
Tearing down inference service trace2333-mistral-trial6-v3-profiler
Service trace2333-mistral-trial6-v3-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.73s
Shutdown handler de-registered
trace2333-mistral-trial6_v3 status is now inactive due to auto deactivation removed underperforming models

Usage Metrics

Latency Metrics