chaiml-elo-alignment-run-3

submission_id: chaiml-elo-alignment-run-3_v44

developer_uid: chai_backend_admin

best_of: 16

celo_rating: 1265.35

display_name: chaiml-elo-alignment-run-3_v44

family_friendly_score: 0.5649468802498638

family_friendly_standard_error: 0.0023630396083560655

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 0.95, 'top_p': 1.0, 'min_p': 0.08, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}

gpu_counts: {'NVIDIA RTX A5000': 1}

is_internal_developer: True

language_model: ChaiML/elo_alignment_run_3

latencies: [{'batch_size': 1, 'throughput': 0.9112412284044937, 'latency_mean': 1.0973161149024964, 'latency_p50': 1.0888301134109497, 'latency_p90': 1.2310284852981568}, {'batch_size': 4, 'throughput': 1.7853778603612922, 'latency_mean': 2.2270393168926237, 'latency_p50': 2.2185938358306885, 'latency_p90': 2.4687954902648928}, {'batch_size': 5, 'throughput': 1.8839640292889537, 'latency_mean': 2.6448255491256716, 'latency_p50': 2.6294543743133545, 'latency_p90': 2.9359061479568482}, {'batch_size': 8, 'throughput': 1.9960545468462831, 'latency_mean': 3.9790811669826507, 'latency_p50': 4.014918565750122, 'latency_p90': 4.429268383979797}, {'batch_size': 10, 'throughput': 2.0263711153047232, 'latency_mean': 4.884935563802719, 'latency_p50': 4.898812174797058, 'latency_p90': 5.480685830116272}, {'batch_size': 12, 'throughput': 2.0225917101281596, 'latency_mean': 5.855033124685288, 'latency_p50': 5.848761796951294, 'latency_p90': 6.75855827331543}, {'batch_size': 15, 'throughput': 2.0373126487935846, 'latency_mean': 7.213669549226761, 'latency_p50': 7.276374697685242, 'latency_p90': 8.004050993919373}]

max_input_tokens: 512

max_output_tokens: 64

model_architecture: LlamaForCausalLM

model_group: ChaiML/elo_alignment_run

model_name: chaiml-elo-alignment-run-3_v44

model_num_parameters: 8030261248.0

model_repo: ChaiML/elo_alignment_run_3

model_size: 8B

num_battles: 6296905

num_wins: 3267309

ranking_group: single

status: inactive

submission_type: basic

throughput_3p7s: 1.99

timestamp: 2024-09-19T21:14:26+00:00

us_pacific_date: 2024-09-19

win_ratio: 0.5188753840180216

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-elo-alignment-run-3-v44-mkmlizer
Waiting for job on chaiml-elo-alignment-run-3-v44-mkmlizer to finish
chaiml-elo-alignment-run-3-v44-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-elo-alignment-run-3-v44-mkmlizer: ║     _____            __           __                                ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║       /___/                                                         ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  belonging to:                                                      ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v44-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-elo-alignment-run-3-v44-mkmlizer: Downloaded to shared memory in 54.652s
chaiml-elo-alignment-run-3-v44-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp0gllgrf4, device:0
chaiml-elo-alignment-run-3-v44-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-elo-alignment-run-3-v44-mkmlizer: quantized model in 28.869s
chaiml-elo-alignment-run-3-v44-mkmlizer: Processed model ChaiML/elo_alignment_run_3 in 83.521s
chaiml-elo-alignment-run-3-v44-mkmlizer: creating bucket guanaco-mkml-models
chaiml-elo-alignment-run-3-v44-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-elo-alignment-run-3-v44-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v44
chaiml-elo-alignment-run-3-v44-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v44/special_tokens_map.json
chaiml-elo-alignment-run-3-v44-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v44/config.json
chaiml-elo-alignment-run-3-v44-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v44/tokenizer_config.json
chaiml-elo-alignment-run-3-v44-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v44/tokenizer.json
chaiml-elo-alignment-run-3-v44-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v44/flywheel_model.0.safetensors
chaiml-elo-alignment-run-3-v44-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:10, 26.30it/s]
Loading 0:   4%|▍         | 12/291 [00:00<00:07, 36.71it/s]
Loading 0:   5%|▌         | 16/291 [00:00<00:08, 33.57it/s]
Loading 0:   7%|▋         | 21/291 [00:00<00:07, 37.34it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:07, 35.23it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:06, 41.58it/s]
Loading 0:  12%|█▏        | 36/291 [00:01<00:10, 24.37it/s]
Loading 0:  14%|█▍        | 41/291 [00:01<00:09, 25.48it/s]
Loading 0:  16%|█▋        | 48/291 [00:01<00:07, 32.55it/s]
Loading 0:  18%|█▊        | 53/291 [00:01<00:07, 33.53it/s]
Loading 0:  20%|█▉        | 57/291 [00:01<00:06, 34.30it/s]
Loading 0:  21%|██        | 61/291 [00:01<00:07, 32.41it/s]
Loading 0:  23%|██▎       | 66/291 [00:02<00:06, 35.02it/s]
Loading 0:  24%|██▍       | 70/291 [00:02<00:06, 33.26it/s]
Loading 0:  25%|██▌       | 74/291 [00:02<00:06, 33.54it/s]
Loading 0:  27%|██▋       | 78/291 [00:02<00:06, 33.91it/s]
Loading 0:  28%|██▊       | 82/291 [00:02<00:08, 24.21it/s]
Loading 0:  29%|██▉       | 85/291 [00:02<00:08, 25.16it/s]
Loading 0:  31%|███       | 90/291 [00:02<00:06, 29.77it/s]
Loading 0:  32%|███▏      | 94/291 [00:03<00:06, 30.15it/s]
Loading 0:  34%|███▍      | 99/291 [00:03<00:05, 33.50it/s]
Loading 0:  35%|███▌      | 103/291 [00:03<00:05, 31.40it/s]
Loading 0:  37%|███▋      | 108/291 [00:03<00:05, 34.19it/s]
Loading 0:  38%|███▊      | 112/291 [00:03<00:05, 32.14it/s]
Loading 0:  40%|███▉      | 116/291 [00:03<00:05, 33.06it/s]
Loading 0:  42%|████▏     | 122/291 [00:03<00:04, 37.47it/s]
Loading 0:  44%|████▎     | 127/291 [00:03<00:04, 34.75it/s]
Loading 0:  46%|████▌     | 133/291 [00:04<00:05, 30.99it/s]
Loading 0:  47%|████▋     | 137/291 [00:04<00:05, 30.67it/s]
Loading 0:  48%|████▊     | 141/291 [00:04<00:05, 28.57it/s]
Loading 0:  51%|█████     | 147/291 [00:04<00:04, 33.59it/s]
Loading 0:  52%|█████▏    | 151/291 [00:04<00:04, 32.54it/s]
Loading 0:  54%|█████▎    | 156/291 [00:04<00:03, 35.23it/s]
Loading 0:  55%|█████▍    | 160/291 [00:04<00:03, 33.40it/s]
Loading 0:  57%|█████▋    | 165/291 [00:05<00:03, 36.47it/s]
Loading 0:  58%|█████▊    | 169/291 [00:05<00:03, 34.67it/s]
Loading 0:  60%|█████▉    | 174/291 [00:05<00:03, 37.41it/s]
Loading 0:  61%|██████    | 178/291 [00:05<00:03, 35.60it/s]
Loading 0:  64%|██████▎   | 185/291 [00:05<00:02, 39.45it/s]
Loading 0:  65%|██████▍   | 189/291 [00:05<00:03, 25.98it/s]
Loading 0:  67%|██████▋   | 194/291 [00:06<00:03, 26.89it/s]
Loading 0:  69%|██████▉   | 201/291 [00:06<00:02, 33.76it/s]
Loading 0:  70%|███████   | 205/291 [00:06<00:02, 32.96it/s]
Loading 0:  72%|███████▏  | 210/291 [00:06<00:02, 35.63it/s]
Loading 0:  74%|███████▎  | 214/291 [00:06<00:02, 34.49it/s]
Loading 0:  75%|███████▌  | 219/291 [00:06<00:01, 37.01it/s]
Loading 0:  77%|███████▋  | 223/291 [00:06<00:01, 34.73it/s]
Loading 0:  78%|███████▊  | 227/291 [00:06<00:01, 34.29it/s]
Loading 0:  79%|███████▉  | 231/291 [00:07<00:01, 34.35it/s]
Loading 0:  81%|████████  | 235/291 [00:07<00:02, 26.30it/s]
Loading 0:  82%|████████▏ | 239/291 [00:07<00:02, 25.62it/s]
Loading 0:  85%|████████▍ | 246/291 [00:07<00:01, 33.25it/s]
Loading 0:  86%|████████▌ | 250/291 [00:07<00:01, 32.46it/s]
Loading 0:  88%|████████▊ | 255/291 [00:07<00:01, 35.54it/s]
Loading 0:  89%|████████▉ | 259/291 [00:07<00:00, 33.95it/s]
Loading 0:  91%|█████████ | 264/291 [00:08<00:00, 36.74it/s]
Loading 0:  92%|█████████▏| 268/291 [00:08<00:00, 34.79it/s]
Loading 0:  94%|█████████▍| 273/291 [00:08<00:00, 37.27it/s]
Loading 0:  95%|█████████▌| 277/291 [00:08<00:00, 34.82it/s]
Loading 0:  97%|█████████▋| 281/291 [00:08<00:00, 34.92it/s]
Loading 0:  98%|█████████▊| 286/291 [00:14<00:01,  2.57it/s]
Loading 0:  99%|█████████▉| 289/291 [00:14<00:00,  3.18it/s]
                                                            
Job chaiml-elo-alignment-run-3-v44-mkmlizer completed after 107.56s with status: succeeded
Stopping job with name chaiml-elo-alignment-run-3-v44-mkmlizer
Pipeline stage MKMLizer completed in 108.54s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.26s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-elo-alignment-run-3-v44
Waiting for inference service chaiml-elo-alignment-run-3-v44 to be ready
Inference service chaiml-elo-alignment-run-3-v44 ready after 30.48172116279602s
Pipeline stage MKMLDeployer completed in 31.36s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.5551397800445557s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.033864974975586s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.7071800231933594s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.8935420513153076s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.1129770278930664s
5 requests
0 failed requests
5th percentile: 1.744452428817749
10th percentile: 1.7817248344421386
20th percentile: 1.856269645690918
30th percentile: 1.9216066360473634
40th percentile: 1.9777358055114747
50th percentile: 2.033864974975586
60th percentile: 2.065509796142578
70th percentile: 2.0971546173095703
80th percentile: 2.2014095783233643
90th percentile: 2.37827467918396
95th percentile: 2.466707229614258
99th percentile: 2.537453269958496
mean time: 2.060540771484375
Pipeline stage StressChecker completed in 12.45s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.85s
Shutdown handler de-registered
chaiml-elo-alignment-run-3_v44 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-elo-alignment-run-3-v44-profiler
Waiting for inference service chaiml-elo-alignment-run-3-v44-profiler to be ready
Inference service chaiml-elo-alignment-run-3-v44-profiler ready after 200.45416903495789s
Pipeline stage MKMLProfilerDeployer completed in 200.79s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-elo-alignmentc4ed4522a5aa21fd6808fbe2f09c603b-deplodzrqn:/code/chaiverse_profiler_1726780862 --namespace tenant-chaiml-guanaco
kubectl exec -it chaiml-elo-alignmentc4ed4522a5aa21fd6808fbe2f09c603b-deplodzrqn --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726780862 && python profiles.py profile --best_of_n 16 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1726780862/summary.json'
kubectl exec -it chaiml-elo-alignmentc4ed4522a5aa21fd6808fbe2f09c603b-deplodzrqn --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726780862/summary.json'
Pipeline stage MKMLProfilerRunner completed in 838.70s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-elo-alignment-run-3-v44-profiler is running
Tearing down inference service chaiml-elo-alignment-run-3-v44-profiler
Service chaiml-elo-alignment-run-3-v44-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.33s
Shutdown handler de-registered