chaiml-elo-alignment-run-3

developer_uid: chai_backend_admin
submission_id: chaiml-elo-alignment-run-3_v19
model_name: chaiml-elo-alignment-run-3_v19
model_group: ChaiML/elo_alignment_run
status: torndown
timestamp: 2024-07-31T02:01:01+00:00
num_battles: 347670
num_wins: 196965
celo_rating: 1253.24
family_friendly_score: 0.0
submission_type: basic
model_repo: ChaiML/elo_alignment_run_3
model_architecture: LlamaForCausalLM
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
model_num_parameters: 8030261248.0
best_of: 16
max_input_tokens: 512
max_output_tokens: 64
display_name: chaiml-elo-alignment-run-3_v19
is_internal_developer: True
language_model: ChaiML/elo_alignment_run_3
model_size: 8B
ranking_group: single
us_pacific_date: 2024-07-30
win_ratio: 0.5665286047113642
generation_params: {'temperature': 0.95, 'top_p': 1.0, 'min_p': 0.08, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64, 'reward_max_token_input': 256}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': '', 'prompt_template': '', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chaiml-elo-alignment-run-3-v19-mkmlizer
Waiting for job on chaiml-elo-alignment-run-3-v19-mkmlizer to finish
chaiml-elo-alignment-run-3-v19-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-elo-alignment-run-3-v19-mkmlizer: ║     _____            __           __                                ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║       /___/                                                         ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  Version: 0.9.7                                                     ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  belonging to:                                                      ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ║                                                                     ║
chaiml-elo-alignment-run-3-v19-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-elo-alignment-run-3-v19-mkmlizer: Downloaded to shared memory in 48.775s
chaiml-elo-alignment-run-3-v19-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpcdfqdv4e, device:0
chaiml-elo-alignment-run-3-v19-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-elo-alignment-run-3-v19-mkmlizer: quantized model in 28.818s
chaiml-elo-alignment-run-3-v19-mkmlizer: Processed model ChaiML/elo_alignment_run_3 in 77.594s
chaiml-elo-alignment-run-3-v19-mkmlizer: creating bucket guanaco-mkml-models
chaiml-elo-alignment-run-3-v19-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-elo-alignment-run-3-v19-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v19
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v19/config.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v19/special_tokens_map.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v19/tokenizer_config.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v19/tokenizer.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-elo-alignment-run-3-v19/flywheel_model.0.safetensors
chaiml-elo-alignment-run-3-v19-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
chaiml-elo-alignment-run-3-v19-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:10, 27.89it/s]
Loading 0:   4%|▍         | 12/291 [00:00<00:07, 37.32it/s]
Loading 0:   5%|▌         | 16/291 [00:00<00:08, 33.83it/s]
Loading 0:   7%|▋         | 21/291 [00:00<00:07, 37.09it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:07, 35.05it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:06, 41.63it/s]
Loading 0:  12%|█▏        | 36/291 [00:01<00:10, 23.97it/s]
Loading 0:  14%|█▍        | 41/291 [00:01<00:09, 25.99it/s]
Loading 0:  16%|█▋        | 48/291 [00:01<00:07, 32.74it/s]
Loading 0:  18%|█▊        | 52/291 [00:01<00:07, 32.36it/s]
Loading 0:  20%|█▉        | 57/291 [00:01<00:06, 35.29it/s]
Loading 0:  21%|██        | 61/291 [00:01<00:06, 34.43it/s]
Loading 0:  23%|██▎       | 66/291 [00:01<00:06, 36.86it/s]
Loading 0:  24%|██▍       | 70/291 [00:02<00:06, 34.80it/s]
Loading 0:  25%|██▌       | 74/291 [00:02<00:06, 34.96it/s]
Loading 0:  27%|██▋       | 78/291 [00:02<00:06, 35.39it/s]
Loading 0:  28%|██▊       | 82/291 [00:02<00:08, 24.67it/s]
Loading 0:  30%|██▉       | 86/291 [00:02<00:07, 27.38it/s]
Loading 0:  31%|███       | 90/291 [00:02<00:06, 29.45it/s]
Loading 0:  32%|███▏      | 94/291 [00:02<00:06, 30.05it/s]
Loading 0:  34%|███▍      | 99/291 [00:03<00:05, 33.81it/s]
Loading 0:  35%|███▌      | 103/291 [00:03<00:05, 32.71it/s]
Loading 0:  37%|███▋      | 108/291 [00:03<00:05, 34.94it/s]
Loading 0:  38%|███▊      | 112/291 [00:03<00:05, 34.01it/s]
Loading 0:  40%|███▉      | 116/291 [00:03<00:05, 34.44it/s]
Loading 0:  42%|████▏     | 122/291 [00:03<00:04, 38.92it/s]
Loading 0:  44%|████▎     | 127/291 [00:03<00:04, 36.85it/s]
Loading 0:  46%|████▌     | 133/291 [00:04<00:05, 29.91it/s]
Loading 0:  47%|████▋     | 137/291 [00:04<00:05, 30.27it/s]
Loading 0:  48%|████▊     | 141/291 [00:04<00:05, 29.10it/s]
Loading 0:  51%|█████     | 147/291 [00:04<00:04, 34.02it/s]
Loading 0:  52%|█████▏    | 151/291 [00:04<00:04, 32.85it/s]
Loading 0:  54%|█████▎    | 156/291 [00:04<00:03, 34.72it/s]
Loading 0:  55%|█████▍    | 160/291 [00:04<00:03, 33.10it/s]
Loading 0:  57%|█████▋    | 165/291 [00:05<00:03, 34.59it/s]
Loading 0:  58%|█████▊    | 169/291 [00:05<00:03, 33.41it/s]
Loading 0:  60%|█████▉    | 174/291 [00:05<00:03, 36.00it/s]
Loading 0:  61%|██████    | 178/291 [00:05<00:03, 34.62it/s]
Loading 0:  63%|██████▎   | 183/291 [00:05<00:02, 38.31it/s]
Loading 0:  64%|██████▍   | 187/291 [00:05<00:03, 26.67it/s]
Loading 0:  66%|██████▌   | 191/291 [00:05<00:03, 28.10it/s]
Loading 0:  67%|██████▋   | 195/291 [00:06<00:03, 27.11it/s]
Loading 0:  69%|██████▉   | 201/291 [00:06<00:02, 32.58it/s]
Loading 0:  70%|███████   | 205/291 [00:06<00:02, 31.58it/s]
Loading 0:  72%|███████▏  | 210/291 [00:06<00:02, 34.25it/s]
Loading 0:  74%|███████▎  | 214/291 [00:06<00:02, 32.94it/s]
Loading 0:  75%|███████▌  | 219/291 [00:06<00:02, 35.47it/s]
Loading 0:  77%|███████▋  | 223/291 [00:06<00:01, 34.02it/s]
Loading 0:  78%|███████▊  | 227/291 [00:06<00:01, 33.61it/s]
Loading 0:  79%|███████▉  | 231/291 [00:07<00:01, 33.65it/s]
Loading 0:  81%|████████  | 235/291 [00:07<00:02, 25.01it/s]
Loading 0:  82%|████████▏ | 239/291 [00:07<00:02, 25.32it/s]
Loading 0:  85%|████████▍ | 246/291 [00:07<00:01, 33.36it/s]
Loading 0:  86%|████████▌ | 250/291 [00:07<00:01, 32.72it/s]
Loading 0:  88%|████████▊ | 255/291 [00:07<00:01, 35.52it/s]
Loading 0:  89%|████████▉ | 259/291 [00:07<00:00, 33.84it/s]
Loading 0:  91%|█████████ | 264/291 [00:08<00:00, 36.69it/s]
Loading 0:  92%|█████████▏| 268/291 [00:08<00:00, 35.14it/s]
Loading 0:  94%|█████████▍| 273/291 [00:08<00:00, 37.88it/s]
Loading 0:  95%|█████████▌| 277/291 [00:08<00:00, 35.89it/s]
Loading 0:  97%|█████████▋| 281/291 [00:08<00:00, 35.86it/s]
Loading 0:  98%|█████████▊| 286/291 [00:14<00:01,  2.59it/s]
Loading 0:  99%|█████████▉| 289/291 [00:14<00:00,  3.23it/s]
                                                            
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-elo-alignment-run-3-v19-mkmlizer:   warnings.warn(
chaiml-elo-alignment-run-3-v19-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-elo-alignment-run-3-v19-mkmlizer:   warnings.warn(
chaiml-elo-alignment-run-3-v19-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-elo-alignment-run-3-v19-mkmlizer:   warnings.warn(
chaiml-elo-alignment-run-3-v19-mkmlizer: 
Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading shards:  50%|█████     | 1/2 [00:05<00:05,  5.46s/it]
Downloading shards: 100%|██████████| 2/2 [00:08<00:00,  3.98s/it]
Downloading shards: 100%|██████████| 2/2 [00:08<00:00,  4.21s/it]
chaiml-elo-alignment-run-3-v19-mkmlizer: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.42it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.98it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.62it/s]
chaiml-elo-alignment-run-3-v19-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
chaiml-elo-alignment-run-3-v19-mkmlizer: Saving duration: 1.284s
chaiml-elo-alignment-run-3-v19-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 13.106s
chaiml-elo-alignment-run-3-v19-mkmlizer: creating bucket guanaco-reward-models
chaiml-elo-alignment-run-3-v19-mkmlizer: Bucket 's3://guanaco-reward-models/' created
chaiml-elo-alignment-run-3-v19-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/chaiml-elo-alignment-run-3-v19_reward
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/chaiml-elo-alignment-run-3-v19_reward/special_tokens_map.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/chaiml-elo-alignment-run-3-v19_reward/config.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/chaiml-elo-alignment-run-3-v19_reward/tokenizer_config.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/chaiml-elo-alignment-run-3-v19_reward/merges.txt
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/chaiml-elo-alignment-run-3-v19_reward/vocab.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/chaiml-elo-alignment-run-3-v19_reward/tokenizer.json
chaiml-elo-alignment-run-3-v19-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/chaiml-elo-alignment-run-3-v19_reward/reward.tensors
Job chaiml-elo-alignment-run-3-v19-mkmlizer completed after 124.47s with status: succeeded
HTTP Request: %s %s "%s %d %s"
Stopping job with name chaiml-elo-alignment-run-3-v19-mkmlizer
Pipeline stage MKMLizer completed in 126.41s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.65s
Running pipeline stage ISVCDeployer
Creating inference service chaiml-elo-alignment-run-3-v19
Waiting for inference service chaiml-elo-alignment-run-3-v19 to be ready
Inference service chaiml-elo-alignment-run-3-v19 ready after 112.16107892990112s
Pipeline stage ISVCDeployer completed in 113.77s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.581631898880005s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.7074759006500244s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.7137529850006104s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.741602897644043s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.7253742218017578s
5 requests
0 failed requests
5th percentile: 1.7087313175201415
10th percentile: 1.7099867343902588
20th percentile: 1.7124975681304933
30th percentile: 1.7160772323608398
40th percentile: 1.7207257270812988
50th percentile: 1.7253742218017578
60th percentile: 1.731865692138672
70th percentile: 1.738357162475586
80th percentile: 1.9096086978912354
90th percentile: 2.2456202983856204
95th percentile: 2.4136260986328124
99th percentile: 2.5480307388305663
mean time: 1.893967580795288
Pipeline stage StressChecker completed in 12.95s
chaiml-elo-alignment-run-3_v19 status is now deployed due to DeploymentManager action
chaiml-elo-alignment-run-3_v19 status is now inactive due to admin request
chaiml-elo-alignment-run-3_v19 status is now torndown due to DeploymentManager action