bbchicago-brt-v1-12-narr_4789

developer_uid: Bbbrun0

submission_id: bbchicago-brt-v1-12-narr_4789_v1

model_name: best_of_4_reward_medium

model_group: BBChicago/Brt_v1.12_narr

status: torndown

timestamp: 2024-08-15T10:33:07+00:00

num_battles: 10906

num_wins: 5084

celo_rating: 1198.38

family_friendly_score: 0.0

submission_type: basic

model_repo: BBChicago/Brt_v1.12_narration_alignment_s4000

model_architecture: LlamaForCausalLM

reward_repo: rirv938/reward_gpt2_medium_preference_24m_e2

model_num_parameters: 8030261248.0

best_of: 4

max_input_tokens: 512

max_output_tokens: 64

display_name: best_of_4_reward_medium

is_internal_developer: False

language_model: BBChicago/Brt_v1.12_narration_alignment_s4000

model_size: 8B

ranking_group: single

us_pacific_date: 2024-08-15

win_ratio: 0.46616541353383456

generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|end_header_id|>', '<|eot_id|>', '\n\n{user_name}'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64, 'reward_max_token_input': 256}

formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}

reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}

Resubmit model

Running pipeline stage MKMLizer
Starting job with name bbchicago-brt-v1-12-narr-4789-v1-mkmlizer
Waiting for job on bbchicago-brt-v1-12-narr-4789-v1-mkmlizer to finish
Stopping job with name bbchicago-brt-v1-12-narr-4789-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name bbchicago-brt-v1-12-narr-4789-v1-mkmlizer
Waiting for job on bbchicago-brt-v1-12-narr-4789-v1-mkmlizer to finish
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║     _____            __           __                                ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║       /___/                                                         ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║                                                                     ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  Version: 0.9.9                                                     ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  https://mk1.ai                                                     ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║                                                                     ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  belonging to:                                                      ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║                                                                     ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  Chai Research Corp.                                                ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ║                                                                     ║
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: Downloaded to shared memory in 68.047s
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp7d3il8tm, device:0
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: quantized model in 29.467s
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: Processed model BBChicago/Brt_v1.12_narration_alignment_s4000 in 97.515s
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: creating bucket guanaco-mkml-models
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4789-v1
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4789-v1/config.json
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4789-v1/special_tokens_map.json
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4789-v1/tokenizer_config.json
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4789-v1/tokenizer.json
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4789-v1/flywheel_model.0.safetensors
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: loading reward model from rirv938/reward_gpt2_medium_preference_24m_e2
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:12, 23.28it/s]
Loading 0:   4%|▍         | 12/291 [00:00<00:07, 38.52it/s]
Loading 0:   6%|▌         | 17/291 [00:00<00:07, 35.65it/s]
Loading 0:   7%|▋         | 21/291 [00:00<00:07, 36.01it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:07, 33.65it/s]
Loading 0:  10%|█         | 30/291 [00:00<00:06, 38.04it/s]
Loading 0:  12%|█▏        | 34/291 [00:01<00:10, 25.34it/s]
Loading 0:  13%|█▎        | 38/291 [00:01<00:09, 26.29it/s]
Loading 0:  14%|█▍        | 42/291 [00:01<00:09, 25.56it/s]
Loading 0:  16%|█▋        | 48/291 [00:01<00:07, 31.23it/s]
Loading 0:  18%|█▊        | 52/291 [00:01<00:07, 30.50it/s]
Loading 0:  20%|█▉        | 57/291 [00:01<00:06, 33.52it/s]
Loading 0:  21%|██        | 61/291 [00:01<00:07, 31.81it/s]
Loading 0:  23%|██▎       | 66/291 [00:02<00:06, 34.24it/s]
Loading 0:  24%|██▍       | 70/291 [00:02<00:06, 32.41it/s]
Loading 0:  25%|██▌       | 74/291 [00:02<00:06, 32.20it/s]
Loading 0:  27%|██▋       | 78/291 [00:02<00:06, 32.32it/s]
Loading 0:  28%|██▊       | 82/291 [00:02<00:09, 23.14it/s]
Loading 0:  29%|██▉       | 85/291 [00:02<00:08, 24.22it/s]
Loading 0:  31%|███       | 90/291 [00:02<00:07, 28.49it/s]
Loading 0:  32%|███▏      | 94/291 [00:03<00:06, 28.83it/s]
Loading 0:  34%|███▍      | 99/291 [00:03<00:05, 32.61it/s]
Loading 0:  35%|███▌      | 103/291 [00:03<00:05, 32.17it/s]
Loading 0:  37%|███▋      | 108/291 [00:03<00:05, 35.68it/s]
Loading 0:  38%|███▊      | 112/291 [00:03<00:05, 33.29it/s]
Loading 0:  40%|███▉      | 116/291 [00:03<00:05, 33.39it/s]
Loading 0:  42%|████▏     | 122/291 [00:03<00:04, 38.05it/s]
Loading 0:  44%|████▎     | 127/291 [00:04<00:04, 35.03it/s]
Loading 0:  46%|████▌     | 133/291 [00:04<00:05, 30.61it/s]
Loading 0:  47%|████▋     | 137/291 [00:04<00:05, 30.30it/s]
Loading 0:  48%|████▊     | 141/291 [00:04<00:05, 27.99it/s]
Loading 0:  51%|█████     | 147/291 [00:04<00:04, 32.50it/s]
Loading 0:  52%|█████▏    | 151/291 [00:04<00:04, 31.01it/s]
Loading 0:  54%|█████▎    | 156/291 [00:04<00:04, 32.32it/s]
Loading 0:  55%|█████▍    | 160/291 [00:05<00:04, 31.27it/s]
Loading 0:  57%|█████▋    | 165/291 [00:05<00:03, 33.71it/s]
Loading 0:  58%|█████▊    | 169/291 [00:05<00:03, 32.06it/s]
Loading 0:  60%|█████▉    | 174/291 [00:05<00:03, 34.29it/s]
Loading 0:  61%|██████    | 178/291 [00:05<00:03, 32.71it/s]
Loading 0:  63%|██████▎   | 183/291 [00:05<00:02, 36.81it/s]
Loading 0:  64%|██████▍   | 187/291 [00:06<00:03, 26.38it/s]
Loading 0:  66%|██████▌   | 191/291 [00:06<00:03, 27.07it/s]
Loading 0:  67%|██████▋   | 195/291 [00:06<00:03, 25.69it/s]
Loading 0:  69%|██████▉   | 201/291 [00:06<00:02, 30.93it/s]
Loading 0:  70%|███████   | 205/291 [00:06<00:02, 30.13it/s]
Loading 0:  72%|███████▏  | 210/291 [00:06<00:02, 32.54it/s]
Loading 0:  74%|███████▎  | 214/291 [00:06<00:02, 31.07it/s]
Loading 0:  75%|███████▌  | 219/291 [00:07<00:02, 33.35it/s]
Loading 0:  77%|███████▋  | 223/291 [00:07<00:02, 30.94it/s]
Loading 0:  78%|███████▊  | 227/291 [00:07<00:02, 30.42it/s]
Loading 0:  79%|███████▉  | 231/291 [00:07<00:01, 30.12it/s]
Loading 0:  81%|████████  | 235/291 [00:07<00:02, 22.19it/s]
Loading 0:  82%|████████▏ | 239/291 [00:07<00:02, 22.15it/s]
Loading 0:  85%|████████▍ | 246/291 [00:08<00:01, 29.21it/s]
Loading 0:  86%|████████▌ | 250/291 [00:08<00:01, 28.70it/s]
Loading 0:  88%|████████▊ | 255/291 [00:08<00:01, 31.02it/s]
Loading 0:  89%|████████▉ | 259/291 [00:08<00:01, 30.21it/s]
Loading 0:  91%|█████████ | 264/291 [00:08<00:00, 33.02it/s]
Loading 0:  92%|█████████▏| 268/291 [00:08<00:00, 32.04it/s]
Loading 0:  94%|█████████▍| 273/291 [00:08<00:00, 34.29it/s]
Loading 0:  95%|█████████▌| 277/291 [00:08<00:00, 32.40it/s]
Loading 0:  97%|█████████▋| 281/291 [00:09<00:00, 31.77it/s]
Loading 0:  98%|█████████▊| 286/291 [00:14<00:01,  2.54it/s]
Loading 0:  99%|█████████▉| 289/291 [00:14<00:00,  3.14it/s]
                                                            
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer:   warnings.warn(
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer:   warnings.warn(
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer:   warnings.warn(
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: Saving duration: 0.374s
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: Processed model rirv938/reward_gpt2_medium_preference_24m_e2 in 5.898s
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: creating bucket guanaco-reward-models
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4789-v1_reward/special_tokens_map.json
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4789-v1_reward/config.json
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4789-v1_reward/merges.txt
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4789-v1_reward/vocab.json
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4789-v1_reward/tokenizer.json
bbchicago-brt-v1-12-narr-4789-v1-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4789-v1_reward/reward.tensors
Job bbchicago-brt-v1-12-narr-4789-v1-mkmlizer completed after 136.61s with status: succeeded
Stopping job with name bbchicago-brt-v1-12-narr-4789-v1-mkmlizer
Pipeline stage MKMLizer completed in 137.69s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.10s
Running pipeline stage ISVCDeployer
Creating inference service bbchicago-brt-v1-12-narr-4789-v1
Waiting for inference service bbchicago-brt-v1-12-narr-4789-v1 to be ready
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Failed to get response for submission chaiml-llama-8b-pairwise_8189_v2: 
Inference service bbchicago-brt-v1-12-narr-4789-v1 ready after 221.38932251930237s
Pipeline stage ISVCDeployer completed in 222.05s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0448977947235107s
Received healthy response to inference request in 1.110868215560913s
Received healthy response to inference request in 1.1104466915130615s
Received healthy response to inference request in 1.0863561630249023s
Received healthy response to inference request in 1.1131465435028076s
5 requests
0 failed requests
5th percentile: 1.091174268722534
10th percentile: 1.095992374420166
20th percentile: 1.1056285858154298
30th percentile: 1.1105309963226317
40th percentile: 1.1106996059417724
50th percentile: 1.110868215560913
60th percentile: 1.111779546737671
70th percentile: 1.1126908779144287
80th percentile: 1.2994967937469484
90th percentile: 1.6721972942352297
95th percentile: 1.85854754447937
99th percentile: 2.0076277446746826
mean time: 1.293143081665039
Pipeline stage StressChecker completed in 7.16s
bbchicago-brt-v1-12-narr_4789_v1 status is now deployed due to DeploymentManager action
bbchicago-brt-v1-12-narr_4789_v1 status is now inactive due to auto deactivation removed underperforming models
bbchicago-brt-v1-12-narr_4789_v1 status is now torndown due to DeploymentManager action