submission_id: rinen0721-llama0920-356_v1
developer_uid: rinen0721
best_of: 8
celo_rating: 1217.12
display_name: rinen0721-llama0920-356_v1
family_friendly_score: 0.0
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
gpu_counts: {'NVIDIA RTX A5000': 1}
is_internal_developer: False
language_model: rinen0721/llama0920-356
latencies: [{'batch_size': 1, 'throughput': 0.8569595351501877, 'latency_mean': 1.1668548285961151, 'latency_p50': 1.1733027696609497, 'latency_p90': 1.2839820861816407}, {'batch_size': 4, 'throughput': 1.846089069449879, 'latency_mean': 2.1560926520824433, 'latency_p50': 2.1383352279663086, 'latency_p90': 2.399861741065979}, {'batch_size': 5, 'throughput': 1.9694829418665918, 'latency_mean': 2.5213099372386933, 'latency_p50': 2.5414326190948486, 'latency_p90': 2.7892786264419556}, {'batch_size': 8, 'throughput': 2.212070027848043, 'latency_mean': 3.5929149389266968, 'latency_p50': 3.569967269897461, 'latency_p90': 4.027084302902222}, {'batch_size': 10, 'throughput': 2.257473300968712, 'latency_mean': 4.391242783069611, 'latency_p50': 4.401536226272583, 'latency_p90': 4.93030617237091}, {'batch_size': 12, 'throughput': 2.306685649999231, 'latency_mean': 5.153833991289138, 'latency_p50': 5.14285683631897, 'latency_p90': 5.855304002761841}, {'batch_size': 15, 'throughput': 2.304660522119978, 'latency_mean': 6.427902450561524, 'latency_p50': 6.427570939064026, 'latency_p90': 7.1679893970489506}]
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: rinen0721/llama0920-356
model_name: rinen0721-llama0920-356_v1
model_num_parameters: 8030261248.0
model_repo: rinen0721/llama0920-356
model_size: 8B
num_battles: 12189
num_wins: 5443
ranking_group: single
status: torndown
submission_type: basic
throughput_3p7s: 2.23
timestamp: 2024-09-20T05:19:41+00:00
us_pacific_date: 2024-09-19
win_ratio: 0.4465501681844286
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rinen0721-llama0920-356-v1-mkmlizer
Waiting for job on rinen0721-llama0920-356-v1-mkmlizer to finish
Failed to get response for submission zonemercy-lexical-nemov8_5966_v2: ('http://zonemercy-lexical-nemov8-5966-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
rinen0721-llama0920-356-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rinen0721-llama0920-356-v1-mkmlizer: ║     _____            __           __                                ║
rinen0721-llama0920-356-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rinen0721-llama0920-356-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rinen0721-llama0920-356-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rinen0721-llama0920-356-v1-mkmlizer: ║       /___/                                                         ║
rinen0721-llama0920-356-v1-mkmlizer: ║                                                                     ║
rinen0721-llama0920-356-v1-mkmlizer: ║  Version: 0.10.1                                                    ║
rinen0721-llama0920-356-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rinen0721-llama0920-356-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rinen0721-llama0920-356-v1-mkmlizer: ║                                                                     ║
rinen0721-llama0920-356-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rinen0721-llama0920-356-v1-mkmlizer: ║  belonging to:                                                      ║
rinen0721-llama0920-356-v1-mkmlizer: ║                                                                     ║
rinen0721-llama0920-356-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rinen0721-llama0920-356-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rinen0721-llama0920-356-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rinen0721-llama0920-356-v1-mkmlizer: ║                                                                     ║
rinen0721-llama0920-356-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rinen0721-llama0920-356-v1-mkmlizer: Downloaded to shared memory in 37.069s
rinen0721-llama0920-356-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpqeny7y9a, device:0
rinen0721-llama0920-356-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
rinen0721-llama0920-356-v1-mkmlizer: quantized model in 25.501s
rinen0721-llama0920-356-v1-mkmlizer: Processed model rinen0721/llama0920-356 in 62.570s
rinen0721-llama0920-356-v1-mkmlizer: creating bucket guanaco-mkml-models
rinen0721-llama0920-356-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rinen0721-llama0920-356-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rinen0721-llama0920-356-v1
rinen0721-llama0920-356-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rinen0721-llama0920-356-v1/config.json
rinen0721-llama0920-356-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rinen0721-llama0920-356-v1/special_tokens_map.json
rinen0721-llama0920-356-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rinen0721-llama0920-356-v1/tokenizer_config.json
rinen0721-llama0920-356-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rinen0721-llama0920-356-v1/tokenizer.json
rinen0721-llama0920-356-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rinen0721-llama0920-356-v1/flywheel_model.0.safetensors
rinen0721-llama0920-356-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 7/291 [00:00<00:05, 54.24it/s]
Loading 0:   8%|▊         | 22/291 [00:00<00:03, 86.38it/s]
Loading 0:  11%|█         | 32/291 [00:00<00:02, 91.33it/s]
Loading 0:  15%|█▍        | 43/291 [00:00<00:02, 85.09it/s]
Loading 0:  20%|█▉        | 58/291 [00:00<00:02, 94.57it/s]
Loading 0:  23%|██▎       | 68/291 [00:00<00:02, 93.94it/s]
Loading 0:  27%|██▋       | 78/291 [00:00<00:02, 95.24it/s]
Loading 0:  30%|███       | 88/291 [00:02<00:08, 23.79it/s]
Loading 0:  33%|███▎      | 97/291 [00:02<00:06, 29.67it/s]
Loading 0:  36%|███▋      | 106/291 [00:02<00:05, 36.57it/s]
Loading 0:  40%|███▉      | 115/291 [00:02<00:04, 43.73it/s]
Loading 0:  43%|████▎     | 124/291 [00:02<00:03, 50.76it/s]
Loading 0:  48%|████▊     | 139/291 [00:02<00:02, 63.98it/s]
Loading 0:  52%|█████▏    | 150/291 [00:02<00:01, 73.05it/s]
Loading 0:  55%|█████▍    | 160/291 [00:02<00:01, 70.30it/s]
Loading 0:  58%|█████▊    | 169/291 [00:02<00:01, 73.41it/s]
Loading 0:  61%|██████    | 178/291 [00:03<00:01, 73.98it/s]
Loading 0:  64%|██████▍   | 187/291 [00:04<00:04, 22.59it/s]
Loading 0:  69%|██████▉   | 202/291 [00:04<00:02, 33.05it/s]
Loading 0:  73%|███████▎  | 212/291 [00:04<00:01, 40.51it/s]
Loading 0:  77%|███████▋  | 223/291 [00:04<00:01, 47.49it/s]
Loading 0:  80%|███████▉  | 232/291 [00:04<00:01, 52.36it/s]
Loading 0:  83%|████████▎ | 241/291 [00:04<00:00, 58.40it/s]
Loading 0:  86%|████████▌ | 250/291 [00:04<00:00, 64.04it/s]
Loading 0:  89%|████████▉ | 260/291 [00:04<00:00, 71.81it/s]
Loading 0:  94%|█████████▍| 274/291 [00:05<00:00, 81.65it/s]
Loading 0:  98%|█████████▊| 284/291 [00:05<00:00, 84.48it/s]
                                                            

Job rinen0721-llama0920-356-v1-mkmlizer completed after 83.72s with status: succeeded
Stopping job with name rinen0721-llama0920-356-v1-mkmlizer
Pipeline stage MKMLizer completed in 84.78s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rinen0721-llama0920-356-v1
Waiting for inference service rinen0721-llama0920-356-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service rinen0721-llama0920-356-v1 ready after 211.56868863105774s
Pipeline stage MKMLDeployer completed in 212.76s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.0098602771759033s
Received healthy response to inference request in 1.6490375995635986s
Received healthy response to inference request in 3.884550094604492s
Received healthy response to inference request in 1.3611114025115967s
Received healthy response to inference request in 1.2780406475067139s
5 requests
0 failed requests
5th percentile: 1.2946547985076904
10th percentile: 1.311268949508667
20th percentile: 1.3444972515106202
30th percentile: 1.418696641921997
40th percentile: 1.5338671207427979
50th percentile: 1.6490375995635986
60th percentile: 2.1933666706085204
70th percentile: 2.737695741653442
80th percentile: 3.184798240661621
90th percentile: 3.5346741676330566
95th percentile: 3.7096121311187744
99th percentile: 3.8495625019073487
mean time: 2.236520004272461
Pipeline stage StressChecker completed in 12.40s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 4.40s
Shutdown handler de-registered
rinen0721-llama0920-356_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rinen0721-llama0920-356-v1-profiler
Waiting for inference service rinen0721-llama0920-356-v1-profiler to be ready
Inference service rinen0721-llama0920-356-v1-profiler ready after 200.454008102417s
Pipeline stage MKMLProfilerDeployer completed in 200.88s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rinen0721-llama0920-356-v1-profiler-predictor-00001-deploynzv2v:/code/chaiverse_profiler_1726810144 --namespace tenant-chaiml-guanaco
kubectl exec -it rinen0721-llama0920-356-v1-profiler-predictor-00001-deploynzv2v --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726810144 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1726810144/summary.json'
kubectl exec -it rinen0721-llama0920-356-v1-profiler-predictor-00001-deploynzv2v --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726810144/summary.json'
Pipeline stage MKMLProfilerRunner completed in 801.78s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rinen0721-llama0920-356-v1-profiler is running
Tearing down inference service rinen0721-llama0920-356-v1-profiler
Service rinen0721-llama0920-356-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.16s
Shutdown handler de-registered
rinen0721-llama0920-356_v1 status is now inactive due to auto deactivation removed underperforming models
rinen0721-llama0920-356_v1 status is now torndown due to DeploymentManager action