chaiml-elo-alignment-run-3

developer_uid: azuruce

submission_id: chaiml-elo-alignment-run-3_v38

model_name: chaiml-elo-alignment-run-3_v3

model_group: ChaiML/elo_alignment_run

status: torndown

timestamp: 2024-09-04T18:23:02+00:00

num_battles: 11237

num_wins: 6065

celo_rating: 1263.64

family_friendly_score: 0.0

submission_type: basic

model_repo: ChaiML/elo_alignment_run_3

model_architecture: LlamaForCausalLM

model_num_parameters: 8030261248.0

best_of: 16

max_input_tokens: 512

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.9116160451712872, 'latency_mean': 1.096855161190033, 'latency_p50': 1.1012250185012817, 'latency_p90': 1.215308928489685}, {'batch_size': 4, 'throughput': 1.7713404575193663, 'latency_mean': 2.2459408617019654, 'latency_p50': 2.247462034225464, 'latency_p90': 2.5130135774612428}, {'batch_size': 5, 'throughput': 1.8459493351272513, 'latency_mean': 2.69961040019989, 'latency_p50': 2.720247268676758, 'latency_p90': 2.9822033405303956}, {'batch_size': 8, 'throughput': 1.972409922643448, 'latency_mean': 4.020453525781631, 'latency_p50': 4.047520875930786, 'latency_p90': 4.49359359741211}, {'batch_size': 10, 'throughput': 2.0107189101849663, 'latency_mean': 4.927948371171952, 'latency_p50': 4.945452809333801, 'latency_p90': 5.65638678073883}, {'batch_size': 12, 'throughput': 2.0070068891882493, 'latency_mean': 5.905208556652069, 'latency_p50': 5.986133694648743, 'latency_p90': 6.616010046005249}, {'batch_size': 15, 'throughput': 1.9948333043891824, 'latency_mean': 7.37212610244751, 'latency_p50': 7.4825087785720825, 'latency_p90': 8.278493857383728}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: chaiml-elo-alignment-run-3_v3

is_internal_developer: True

language_model: ChaiML/elo_alignment_run_3

model_size: 8B

ranking_group: single

throughput_3p7s: 1.97

us_pacific_date: 2024-09-04

win_ratio: 0.5397348046631664

generation_params: {'temperature': 0.95, 'top_p': 1.0, 'min_p': 0.08, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-elo-alignment-run-3-v38-mkmlizer
Waiting for job on chaiml-elo-alignment-run-3-v38-mkmlizer to finish
chaiml-elo-alignment-run-3-v38-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-elo-alignment-run-3-v38-mkmlizer: ║     _____            __           __                                ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║       /___/                                                         ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  Version: 0.10.1                                                    ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  belonging to:                                                      ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v38-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-elo-alignment-run-3-v38-mkmlizer: quantized model in 29.287s
chaiml-elo-alignment-run-3-v38-mkmlizer: Processed model ChaiML/elo_alignment_run_3 in 324.160s
chaiml-elo-alignment-run-3-v38-mkmlizer: creating bucket guanaco-mkml-models
chaiml-elo-alignment-run-3-v38-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-elo-alignment-run-3-v38-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v38
chaiml-elo-alignment-run-3-v38-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v38/config.json
chaiml-elo-alignment-run-3-v38-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v38/special_tokens_map.json
chaiml-elo-alignment-run-3-v38-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v38/tokenizer.json
chaiml-elo-alignment-run-3-v38-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v38/flywheel_model.0.safetensors
chaiml-elo-alignment-run-3-v38-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:11, 25.99it/s]
Loading 0:   4%|▍         | 12/291 [00:00<00:07, 36.07it/s]
Loading 0:   5%|▌         | 16/291 [00:00<00:08, 33.91it/s]
Loading 0:   7%|▋         | 21/291 [00:00<00:07, 35.83it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:08, 32.40it/s]
Loading 0:  10%|█         | 30/291 [00:00<00:07, 36.36it/s]
Loading 0:  12%|█▏        | 34/291 [00:01<00:10, 25.41it/s]
Loading 0:  13%|█▎        | 38/291 [00:01<00:09, 25.99it/s]
Loading 0:  14%|█▍        | 41/291 [00:01<00:10, 23.38it/s]
Loading 0:  16%|█▋        | 48/291 [00:01<00:07, 31.70it/s]
Loading 0:  18%|█▊        | 52/291 [00:01<00:07, 31.20it/s]
Loading 0:  20%|█▉        | 57/291 [00:01<00:06, 33.81it/s]
Loading 0:  21%|██        | 61/291 [00:01<00:06, 32.90it/s]
Loading 0:  23%|██▎       | 66/291 [00:02<00:06, 35.98it/s]
Loading 0:  24%|██▍       | 70/291 [00:02<00:06, 33.84it/s]
Loading 0:  25%|██▌       | 74/291 [00:02<00:06, 34.24it/s]
Loading 0:  27%|██▋       | 78/291 [00:02<00:06, 33.20it/s]
Loading 0:  28%|██▊       | 82/291 [00:02<00:08, 23.59it/s]
Loading 0:  29%|██▉       | 85/291 [00:02<00:08, 24.17it/s]
Loading 0:  31%|███       | 90/291 [00:02<00:07, 27.55it/s]
Loading 0:  32%|███▏      | 94/291 [00:03<00:07, 27.83it/s]
Loading 0:  34%|███▍      | 99/291 [00:03<00:06, 30.51it/s]
Loading 0:  35%|███▌      | 103/291 [00:03<00:06, 29.46it/s]
Loading 0:  37%|███▋      | 108/291 [00:03<00:05, 32.34it/s]
Loading 0:  38%|███▊      | 112/291 [00:03<00:05, 31.03it/s]
Loading 0:  40%|███▉      | 116/291 [00:03<00:05, 30.98it/s]
Loading 0:  42%|████▏     | 122/291 [00:03<00:04, 36.10it/s]
Loading 0:  44%|████▎     | 127/291 [00:04<00:04, 35.05it/s]
Loading 0:  46%|████▌     | 133/291 [00:04<00:05, 30.07it/s]
Loading 0:  47%|████▋     | 137/291 [00:04<00:05, 30.36it/s]
Loading 0:  48%|████▊     | 141/291 [00:04<00:05, 27.79it/s]
Loading 0:  51%|█████     | 147/291 [00:04<00:04, 32.52it/s]
Loading 0:  52%|█████▏    | 151/291 [00:04<00:04, 31.26it/s]
Loading 0:  54%|█████▎    | 156/291 [00:05<00:04, 33.05it/s]
Loading 0:  55%|█████▍    | 160/291 [00:05<00:04, 30.88it/s]
Loading 0:  57%|█████▋    | 165/291 [00:05<00:03, 32.99it/s]
Loading 0:  58%|█████▊    | 169/291 [00:05<00:03, 31.49it/s]
Loading 0:  60%|█████▉    | 174/291 [00:05<00:03, 33.61it/s]
Loading 0:  61%|██████    | 178/291 [00:05<00:03, 33.09it/s]
Loading 0:  63%|██████▎   | 184/291 [00:05<00:02, 39.04it/s]
Loading 0:  65%|██████▍   | 189/291 [00:06<00:04, 23.82it/s]
Loading 0:  67%|██████▋   | 194/291 [00:06<00:03, 25.54it/s]
Loading 0:  69%|██████▉   | 201/291 [00:06<00:02, 31.64it/s]
Loading 0:  70%|███████   | 205/291 [00:06<00:02, 31.11it/s]
Loading 0:  72%|███████▏  | 210/291 [00:06<00:02, 32.75it/s]
Loading 0:  74%|███████▎  | 214/291 [00:06<00:02, 31.38it/s]
Loading 0:  75%|███████▌  | 219/291 [00:07<00:02, 34.50it/s]
Loading 0:  77%|███████▋  | 223/291 [00:07<00:02, 33.27it/s]
Loading 0:  78%|███████▊  | 227/291 [00:07<00:01, 34.24it/s]
Loading 0:  79%|███████▉  | 231/291 [00:07<00:01, 33.39it/s]
Loading 0:  81%|████████  | 235/291 [00:07<00:02, 26.16it/s]
Loading 0:  82%|████████▏ | 239/291 [00:07<00:02, 25.86it/s]
Loading 0:  85%|████████▍ | 246/291 [00:07<00:01, 32.34it/s]
Loading 0:  86%|████████▌ | 250/291 [00:08<00:01, 32.04it/s]
Loading 0:  88%|████████▊ | 255/291 [00:08<00:01, 34.54it/s]
Loading 0:  89%|████████▉ | 259/291 [00:08<00:00, 32.78it/s]
Loading 0:  91%|█████████ | 264/291 [00:08<00:00, 35.50it/s]
Loading 0:  92%|█████████▏| 268/291 [00:08<00:00, 33.29it/s]
Loading 0:  94%|█████████▍| 273/291 [00:08<00:00, 36.19it/s]
Loading 0:  95%|█████████▌| 277/291 [00:08<00:00, 33.68it/s]
Loading 0:  97%|█████████▋| 281/291 [00:08<00:00, 33.62it/s]
Loading 0:  98%|█████████▊| 286/291 [00:14<00:01,  2.55it/s]
Loading 0:  99%|█████████▉| 289/291 [00:14<00:00,  3.17it/s]
                                                            
Job chaiml-elo-alignment-run-3-v38-mkmlizer completed after 343.36s with status: succeeded
Stopping job with name chaiml-elo-alignment-run-3-v38-mkmlizer
Pipeline stage MKMLizer completed in 345.35s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.22s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-elo-alignment-run-3-v38
Waiting for inference service chaiml-elo-alignment-run-3-v38 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service chaiml-elo-alignment-run-3-v38 ready after 160.99823141098022s
Pipeline stage MKMLDeployer completed in 161.36s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0966172218322754s
Received healthy response to inference request in 2.1170713901519775s
Received healthy response to inference request in 1.5158860683441162s
Received healthy response to inference request in 1.9091026782989502s
Received healthy response to inference request in 1.9051008224487305s
5 requests
0 failed requests
5th percentile: 1.593729019165039
10th percentile: 1.671571969985962
20th percentile: 1.8272578716278076
30th percentile: 1.9059011936187744
40th percentile: 1.9075019359588623
50th percentile: 1.9091026782989502
60th percentile: 1.9841084957122803
70th percentile: 2.0591143131256104
80th percentile: 2.1007080554962156
90th percentile: 2.1088897228240966
95th percentile: 2.1129805564880373
99th percentile: 2.1162532234191893
mean time: 1.90875563621521
Pipeline stage StressChecker completed in 10.23s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
starting trigger_guanaco_pipeline %s
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.55s
chaiml-elo-alignment-run-3_v38 status is now deployed due to DeploymentManager action
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-elo-alignment-run-3-v38-profiler
Waiting for inference service chaiml-elo-alignment-run-3-v38-profiler to be ready
Inference service chaiml-elo-alignment-run-3-v38-profiler ready after 130.3660352230072s
Pipeline stage MKMLProfilerDeployer completed in 130.73s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-elo-alignmentf5e1d7b23aeaffd715e971faa0401b03-deplo2rsr6:/code/chaiverse_profiler_1725474878 --namespace tenant-chaiml-guanaco
kubectl exec -it chaiml-elo-alignmentf5e1d7b23aeaffd715e971faa0401b03-deplo2rsr6 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1725474878 && python profiles.py profile --best_of_n 16 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725474878/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-elo-alignment-run-3-v38-profiler is running
Tearing down inference service chaiml-elo-alignment-run-3-v38-profiler
Service chaiml-elo-alignment-run-3-v38-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.64s
Shutdown handler unregistered and original handlers restored
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-elo-alignment-run-3-v38-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.77s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-elo-alignment-run-3-v38-profiler
Waiting for inference service chaiml-elo-alignment-run-3-v38-profiler to be ready
Inference service chaiml-elo-alignment-run-3-v38-profiler ready after 150.4559235572815s
Pipeline stage MKMLProfilerDeployer completed in 150.81s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-elo-alignmentf5e1d7b23aeaffd715e971faa0401b03-deplodpjjn:/code/chaiverse_profiler_1725476141 --namespace tenant-chaiml-guanaco
kubectl exec -it chaiml-elo-alignmentf5e1d7b23aeaffd715e971faa0401b03-deplodpjjn --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1725476141 && python profiles.py profile --best_of_n 16 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725476141/summary.json'
kubectl exec -it chaiml-elo-alignmentf5e1d7b23aeaffd715e971faa0401b03-deplodpjjn --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1725476141/summary.json'
Pipeline stage MKMLProfilerRunner completed in 849.92s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-elo-alignment-run-3-v38-profiler is running
Tearing down inference service chaiml-elo-alignment-run-3-v38-profiler
Service chaiml-elo-alignment-run-3-v38-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.64s
Shutdown handler de-registered
chaiml-elo-alignment-run-3_v38 status is now inactive due to auto deactivation removed underperforming models
chaiml-elo-alignment-run-3_v38 status is now torndown due to DeploymentManager action