bbchicago-test-113k-dpo-_5640

developer_uid: Bbbrun0

submission_id: bbchicago-test-113k-dpo-_5640_v5

model_name: auto

model_group: BBChicago/test_113k_dpo_

status: torndown

timestamp: 2024-08-14T02:30:32+00:00

num_battles: 12031

num_wins: 6446

celo_rating: 1241.04

family_friendly_score: 0.0

submission_type: basic

model_repo: BBChicago/test_113k_dpo_steps3000

model_architecture: LlamaForCausalLM

reward_repo: Jellywibble/gpt2_xl_pairwise_89m_step_347634

model_num_parameters: 8030261248.0

best_of: 16

max_input_tokens: 512

max_output_tokens: 64

display_name: auto

is_internal_developer: False

language_model: BBChicago/test_113k_dpo_steps3000

model_size: 8B

ranking_group: single

us_pacific_date: 2024-08-13

win_ratio: 0.5357825617155682

generation_params: {'temperature': 0.95, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|end_header_id|>', '<|eot_id|>', '\n\n{user_name}'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64, 'reward_max_token_input': 256}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}

Resubmit model

Running pipeline stage MKMLizer
Starting job with name bbchicago-test-113k-dpo-5640-v5-mkmlizer
Waiting for job on bbchicago-test-113k-dpo-5640-v5-mkmlizer to finish
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║     _____            __           __                                ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║       /___/                                                         ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║                                                                     ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  Version: 0.9.9                                                     ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  https://mk1.ai                                                     ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║                                                                     ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  The license key for the current software has been verified as      ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  belonging to:                                                      ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║                                                                     ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  Chai Research Corp.                                                ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║                                                                     ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Downloaded to shared memory in 39.140s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp0v5gtflq, device:0
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
bbchicago-test-113k-dpo-5640-v5-mkmlizer: quantized model in 29.520s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Processed model BBChicago/test_113k_dpo_steps3000 in 68.660s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: creating bucket guanaco-mkml-models
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/bbchicago-test-113k-dpo-5640-v5/tokenizer.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/bbchicago-test-113k-dpo-5640-v5/flywheel_model.0.safetensors
bbchicago-test-113k-dpo-5640-v5-mkmlizer: loading reward model from Jellywibble/gpt2_xl_pairwise_89m_step_347634
bbchicago-test-113k-dpo-5640-v5-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:10, 26.19it/s]
Loading 0:   4%|▍         | 12/291 [00:00<00:07, 35.05it/s]
Loading 0:   5%|▌         | 16/291 [00:00<00:08, 32.82it/s]
Loading 0:   7%|▋         | 21/291 [00:00<00:07, 36.16it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:07, 33.51it/s]
Loading 0:  10%|█         | 30/291 [00:00<00:06, 37.41it/s]
Loading 0:  12%|█▏        | 34/291 [00:01<00:10, 24.94it/s]
Loading 0:  13%|█▎        | 38/291 [00:01<00:09, 26.39it/s]
Loading 0:  14%|█▍        | 42/291 [00:01<00:09, 25.32it/s]
Loading 0:  16%|█▋        | 48/291 [00:01<00:07, 30.70it/s]
Loading 0:  18%|█▊        | 52/291 [00:01<00:07, 30.21it/s]
Loading 0:  20%|█▉        | 57/291 [00:01<00:07, 32.57it/s]
Loading 0:  21%|██        | 61/291 [00:01<00:07, 31.63it/s]
Loading 0:  23%|██▎       | 66/291 [00:02<00:06, 33.87it/s]
Loading 0:  24%|██▍       | 70/291 [00:02<00:06, 32.43it/s]
Loading 0:  25%|██▌       | 74/291 [00:02<00:06, 31.57it/s]
Loading 0:  27%|██▋       | 78/291 [00:02<00:06, 31.52it/s]
Loading 0:  28%|██▊       | 82/291 [00:02<00:09, 22.66it/s]
Loading 0:  29%|██▉       | 85/291 [00:02<00:08, 23.92it/s]
Loading 0:  31%|███       | 90/291 [00:03<00:07, 28.00it/s]
Loading 0:  32%|███▏      | 94/291 [00:03<00:06, 28.34it/s]
Loading 0:  34%|███▍      | 99/291 [00:03<00:06, 31.34it/s]
Loading 0:  35%|███▌      | 103/291 [00:03<00:06, 30.14it/s]
Loading 0:  37%|███▋      | 108/291 [00:03<00:05, 32.88it/s]
Loading 0:  38%|███▊      | 112/291 [00:03<00:05, 31.70it/s]
Loading 0:  40%|███▉      | 116/291 [00:03<00:05, 31.96it/s]
Loading 0:  42%|████▏     | 122/291 [00:03<00:04, 35.89it/s]
Loading 0:  44%|████▎     | 127/291 [00:04<00:05, 32.77it/s]
Loading 0:  46%|████▌     | 133/291 [00:04<00:05, 28.09it/s]
Loading 0:  47%|████▋     | 137/291 [00:04<00:05, 28.36it/s]
Loading 0:  48%|████▊     | 140/291 [00:04<00:06, 24.55it/s]
Loading 0:  50%|████▉     | 145/291 [00:04<00:04, 29.47it/s]
Loading 0:  51%|█████     | 149/291 [00:05<00:05, 27.55it/s]
Loading 0:  54%|█████▎    | 156/291 [00:05<00:03, 33.93it/s]
Loading 0:  55%|█████▍    | 160/291 [00:05<00:04, 32.25it/s]
Loading 0:  57%|█████▋    | 165/291 [00:05<00:03, 34.70it/s]
Loading 0:  58%|█████▊    | 169/291 [00:05<00:03, 33.39it/s]
Loading 0:  60%|█████▉    | 174/291 [00:05<00:03, 35.10it/s]
Loading 0:  61%|██████    | 178/291 [00:05<00:03, 33.42it/s]
Loading 0:  63%|██████▎   | 184/291 [00:05<00:02, 39.17it/s]
Loading 0:  65%|██████▍   | 189/291 [00:06<00:04, 23.73it/s]
Loading 0:  67%|██████▋   | 194/291 [00:06<00:03, 25.42it/s]
Loading 0:  69%|██████▉   | 201/291 [00:06<00:02, 31.73it/s]
Loading 0:  70%|███████   | 205/291 [00:06<00:02, 31.28it/s]
Loading 0:  72%|███████▏  | 210/291 [00:06<00:02, 33.89it/s]
Loading 0:  74%|███████▎  | 214/291 [00:06<00:02, 32.70it/s]
Loading 0:  75%|███████▌  | 219/291 [00:07<00:02, 34.96it/s]
Loading 0:  77%|███████▋  | 223/291 [00:07<00:02, 33.42it/s]
Loading 0:  78%|███████▊  | 227/291 [00:07<00:01, 33.04it/s]
Loading 0:  79%|███████▉  | 231/291 [00:07<00:01, 33.07it/s]
Loading 0:  81%|████████  | 235/291 [00:07<00:02, 24.73it/s]
Loading 0:  82%|████████▏ | 239/291 [00:07<00:02, 24.77it/s]
Loading 0:  85%|████████▍ | 246/291 [00:08<00:01, 32.51it/s]
Loading 0:  86%|████████▌ | 250/291 [00:08<00:01, 32.11it/s]
Loading 0:  88%|████████▊ | 255/291 [00:08<00:01, 34.20it/s]
Loading 0:  89%|████████▉ | 259/291 [00:08<00:00, 32.97it/s]
Loading 0:  91%|█████████ | 264/291 [00:08<00:00, 35.33it/s]
Loading 0:  92%|█████████▏| 268/291 [00:08<00:00, 33.98it/s]
Loading 0:  94%|█████████▍| 273/291 [00:08<00:00, 35.51it/s]
Loading 0:  95%|█████████▌| 277/291 [00:08<00:00, 33.70it/s]
Loading 0:  97%|█████████▋| 281/291 [00:09<00:00, 33.60it/s]
Loading 0:  98%|█████████▊| 286/291 [00:14<00:01,  2.55it/s]
Loading 0:  99%|█████████▉| 289/291 [00:14<00:00,  3.16it/s]
                                                            
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-test-113k-dpo-5640-v5-mkmlizer:   warnings.warn(
bbchicago-test-113k-dpo-5640-v5-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-test-113k-dpo-5640-v5-mkmlizer:   warnings.warn(
bbchicago-test-113k-dpo-5640-v5-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-test-113k-dpo-5640-v5-mkmlizer:   warnings.warn(
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Saving duration: 1.370s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Processed model Jellywibble/gpt2_xl_pairwise_89m_step_347634 in 11.580s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: creating bucket guanaco-reward-models
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Bucket 's3://guanaco-reward-models/' created
bbchicago-test-113k-dpo-5640-v5-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/config.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/special_tokens_map.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/tokenizer_config.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/merges.txt
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/vocab.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/tokenizer.json
Job bbchicago-test-113k-dpo-5640-v5-mkmlizer completed after 111.94s with status: succeeded
Stopping job with name bbchicago-test-113k-dpo-5640-v5-mkmlizer
Pipeline stage MKMLizer completed in 113.58s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service bbchicago-test-113k-dpo-5640-v5
Waiting for inference service bbchicago-test-113k-dpo-5640-v5 to be ready
Inference service bbchicago-test-113k-dpo-5640-v5 ready after 279.8037040233612s
Pipeline stage ISVCDeployer completed in 281.78s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2695934772491455s
Received healthy response to inference request in 1.4301669597625732s
Received healthy response to inference request in 1.4386160373687744s
Received healthy response to inference request in 1.433732032775879s
Received healthy response to inference request in 1.4439115524291992s
5 requests
0 failed requests
5th percentile: 1.4308799743652343
10th percentile: 1.4315929889678956
20th percentile: 1.4330190181732179
30th percentile: 1.434708833694458
40th percentile: 1.4366624355316162
50th percentile: 1.4386160373687744
60th percentile: 1.4407342433929444
70th percentile: 1.4428524494171142
80th percentile: 1.6090479373931887
90th percentile: 1.9393207073211671
95th percentile: 2.104457092285156
99th percentile: 2.236566200256348
mean time: 1.6032040119171143
Pipeline stage StressChecker completed in 8.80s
bbchicago-test-113k-dpo-_5640_v5 status is now deployed due to DeploymentManager action
bbchicago-test-113k-dpo-_5640_v5 status is now inactive due to auto deactivation removed underperforming models
bbchicago-test-113k-dpo-_5640_v5 status is now torndown due to DeploymentManager action