submission_id: bbchicago-test-113k-dpo-_5640_v5
developer_uid: Bbbrun0
alignment_samples: 12031
alignment_score: 1.572880519134279
best_of: 16
celo_rating: 1241.1
display_name: auto
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.95, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|end_header_id|>', '<|eot_id|>', '\n\n{user_name}'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64, 'reward_max_token_input': 256}
is_internal_developer: False
language_model: BBChicago/test_113k_dpo_steps3000
max_input_tokens: 512
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: BBChicago/test_113k_dpo_
model_name: auto
model_num_parameters: 8030261248.0
model_repo: BBChicago/test_113k_dpo_steps3000
model_size: 8B
num_battles: 12031
num_wins: 6446
propriety_score: 0.7144203581526861
propriety_total_count: 1061.0
ranking_group: single
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: Jellywibble/gpt2_xl_pairwise_89m_step_347634
status: torndown
submission_type: basic
timestamp: 2024-08-14T02:30:32+00:00
us_pacific_date: 2024-08-13
win_ratio: 0.5357825617155682
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name bbchicago-test-113k-dpo-5640-v5-mkmlizer
Waiting for job on bbchicago-test-113k-dpo-5640-v5-mkmlizer to finish
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ _____ __ __ ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ /___/ ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ Version: 0.9.9 ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ https://mk1.ai ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ The license key for the current software has been verified as ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ belonging to: ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ Chai Research Corp. ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ║ ║
bbchicago-test-113k-dpo-5640-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Downloaded to shared memory in 39.140s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp0v5gtflq, device:0
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
bbchicago-test-113k-dpo-5640-v5-mkmlizer: quantized model in 29.520s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Processed model BBChicago/test_113k_dpo_steps3000 in 68.660s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: creating bucket guanaco-mkml-models
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/bbchicago-test-113k-dpo-5640-v5/tokenizer.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/bbchicago-test-113k-dpo-5640-v5/flywheel_model.0.safetensors
bbchicago-test-113k-dpo-5640-v5-mkmlizer: loading reward model from Jellywibble/gpt2_xl_pairwise_89m_step_347634
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 2%|▏ | 5/291 [00:00<00:10, 26.19it/s] Loading 0: 4%|▍ | 12/291 [00:00<00:07, 35.05it/s] Loading 0: 5%|▌ | 16/291 [00:00<00:08, 32.82it/s] Loading 0: 7%|▋ | 21/291 [00:00<00:07, 36.16it/s] Loading 0: 9%|▊ | 25/291 [00:00<00:07, 33.51it/s] Loading 0: 10%|█ | 30/291 [00:00<00:06, 37.41it/s] Loading 0: 12%|█▏ | 34/291 [00:01<00:10, 24.94it/s] Loading 0: 13%|█▎ | 38/291 [00:01<00:09, 26.39it/s] Loading 0: 14%|█▍ | 42/291 [00:01<00:09, 25.32it/s] Loading 0: 16%|█▋ | 48/291 [00:01<00:07, 30.70it/s] Loading 0: 18%|█▊ | 52/291 [00:01<00:07, 30.21it/s] Loading 0: 20%|█▉ | 57/291 [00:01<00:07, 32.57it/s] Loading 0: 21%|██ | 61/291 [00:01<00:07, 31.63it/s] Loading 0: 23%|██▎ | 66/291 [00:02<00:06, 33.87it/s] Loading 0: 24%|██▍ | 70/291 [00:02<00:06, 32.43it/s] Loading 0: 25%|██▌ | 74/291 [00:02<00:06, 31.57it/s] Loading 0: 27%|██▋ | 78/291 [00:02<00:06, 31.52it/s] Loading 0: 28%|██▊ | 82/291 [00:02<00:09, 22.66it/s] Loading 0: 29%|██▉ | 85/291 [00:02<00:08, 23.92it/s] Loading 0: 31%|███ | 90/291 [00:03<00:07, 28.00it/s] Loading 0: 32%|███▏ | 94/291 [00:03<00:06, 28.34it/s] Loading 0: 34%|███▍ | 99/291 [00:03<00:06, 31.34it/s] Loading 0: 35%|███▌ | 103/291 [00:03<00:06, 30.14it/s] Loading 0: 37%|███▋ | 108/291 [00:03<00:05, 32.88it/s] Loading 0: 38%|███▊ | 112/291 [00:03<00:05, 31.70it/s] Loading 0: 40%|███▉ | 116/291 [00:03<00:05, 31.96it/s] Loading 0: 42%|████▏ | 122/291 [00:03<00:04, 35.89it/s] Loading 0: 44%|████▎ | 127/291 [00:04<00:05, 32.77it/s] Loading 0: 46%|████▌ | 133/291 [00:04<00:05, 28.09it/s] Loading 0: 47%|████▋ | 137/291 [00:04<00:05, 28.36it/s] Loading 0: 48%|████▊ | 140/291 [00:04<00:06, 24.55it/s] Loading 0: 50%|████▉ | 145/291 [00:04<00:04, 29.47it/s] Loading 0: 51%|█████ | 149/291 [00:05<00:05, 27.55it/s] Loading 0: 54%|█████▎ | 156/291 [00:05<00:03, 33.93it/s] Loading 0: 55%|█████▍ | 160/291 [00:05<00:04, 32.25it/s] Loading 0: 57%|█████▋ | 165/291 [00:05<00:03, 34.70it/s] Loading 0: 58%|█████▊ | 169/291 [00:05<00:03, 33.39it/s] Loading 0: 60%|█████▉ | 174/291 [00:05<00:03, 35.10it/s] Loading 0: 61%|██████ | 178/291 [00:05<00:03, 33.42it/s] Loading 0: 63%|██████▎ | 184/291 [00:05<00:02, 39.17it/s] Loading 0: 65%|██████▍ | 189/291 [00:06<00:04, 23.73it/s] Loading 0: 67%|██████▋ | 194/291 [00:06<00:03, 25.42it/s] Loading 0: 69%|██████▉ | 201/291 [00:06<00:02, 31.73it/s] Loading 0: 70%|███████ | 205/291 [00:06<00:02, 31.28it/s] Loading 0: 72%|███████▏ | 210/291 [00:06<00:02, 33.89it/s] Loading 0: 74%|███████▎ | 214/291 [00:06<00:02, 32.70it/s] Loading 0: 75%|███████▌ | 219/291 [00:07<00:02, 34.96it/s] Loading 0: 77%|███████▋ | 223/291 [00:07<00:02, 33.42it/s] Loading 0: 78%|███████▊ | 227/291 [00:07<00:01, 33.04it/s] Loading 0: 79%|███████▉ | 231/291 [00:07<00:01, 33.07it/s] Loading 0: 81%|████████ | 235/291 [00:07<00:02, 24.73it/s] Loading 0: 82%|████████▏ | 239/291 [00:07<00:02, 24.77it/s] Loading 0: 85%|████████▍ | 246/291 [00:08<00:01, 32.51it/s] Loading 0: 86%|████████▌ | 250/291 [00:08<00:01, 32.11it/s] Loading 0: 88%|████████▊ | 255/291 [00:08<00:01, 34.20it/s] Loading 0: 89%|████████▉ | 259/291 [00:08<00:00, 32.97it/s] Loading 0: 91%|█████████ | 264/291 [00:08<00:00, 35.33it/s] Loading 0: 92%|█████████▏| 268/291 [00:08<00:00, 33.98it/s] Loading 0: 94%|█████████▍| 273/291 [00:08<00:00, 35.51it/s] Loading 0: 95%|█████████▌| 277/291 [00:08<00:00, 33.70it/s] Loading 0: 97%|█████████▋| 281/291 [00:09<00:00, 33.60it/s] Loading 0: 98%|█████████▊| 286/291 [00:14<00:01, 2.55it/s] Loading 0: 99%|█████████▉| 289/291 [00:14<00:00, 3.16it/s] /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-test-113k-dpo-5640-v5-mkmlizer: warnings.warn(
bbchicago-test-113k-dpo-5640-v5-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-test-113k-dpo-5640-v5-mkmlizer: warnings.warn(
bbchicago-test-113k-dpo-5640-v5-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-test-113k-dpo-5640-v5-mkmlizer: warnings.warn(
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Saving duration: 1.370s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Processed model Jellywibble/gpt2_xl_pairwise_89m_step_347634 in 11.580s
bbchicago-test-113k-dpo-5640-v5-mkmlizer: creating bucket guanaco-reward-models
bbchicago-test-113k-dpo-5640-v5-mkmlizer: Bucket 's3://guanaco-reward-models/' created
bbchicago-test-113k-dpo-5640-v5-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/config.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/special_tokens_map.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/tokenizer_config.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/merges.txt
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/vocab.json
bbchicago-test-113k-dpo-5640-v5-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/bbchicago-test-113k-dpo-5640-v5_reward/tokenizer.json
Job bbchicago-test-113k-dpo-5640-v5-mkmlizer completed after 111.94s with status: succeeded
Stopping job with name bbchicago-test-113k-dpo-5640-v5-mkmlizer
Pipeline stage MKMLizer completed in 113.58s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service bbchicago-test-113k-dpo-5640-v5
Waiting for inference service bbchicago-test-113k-dpo-5640-v5 to be ready
Inference service bbchicago-test-113k-dpo-5640-v5 ready after 279.8037040233612s
Pipeline stage ISVCDeployer completed in 281.78s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2695934772491455s
Received healthy response to inference request in 1.4301669597625732s
Received healthy response to inference request in 1.4386160373687744s
Received healthy response to inference request in 1.433732032775879s
Received healthy response to inference request in 1.4439115524291992s
5 requests
0 failed requests
5th percentile: 1.4308799743652343
10th percentile: 1.4315929889678956
20th percentile: 1.4330190181732179
30th percentile: 1.434708833694458
40th percentile: 1.4366624355316162
50th percentile: 1.4386160373687744
60th percentile: 1.4407342433929444
70th percentile: 1.4428524494171142
80th percentile: 1.6090479373931887
90th percentile: 1.9393207073211671
95th percentile: 2.104457092285156
99th percentile: 2.236566200256348
mean time: 1.6032040119171143
Pipeline stage StressChecker completed in 8.80s
bbchicago-test-113k-dpo-_5640_v5 status is now deployed due to DeploymentManager action
bbchicago-test-113k-dpo-_5640_v5 status is now inactive due to auto deactivation removed underperforming models
bbchicago-test-113k-dpo-_5640_v5 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics