submission_id: bbchicago-brt-v1-12-narr_4893_v2
developer_uid: Bbbrun0
best_of: 4
celo_rating: 1195.95
display_name: best_of_4_reward_medium
family_friendly_score: 0.0
formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|end_header_id|>', '<|eot_id|>', '\n\n{user_name}'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64, 'reward_max_token_input': 256}
is_internal_developer: False
language_model: BBChicago/Brt_v1.12_narration_alignment_s2500
max_input_tokens: 512
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: BBChicago/Brt_v1.12_narr
model_name: best_of_4_reward_medium
model_num_parameters: 8030261248.0
model_repo: BBChicago/Brt_v1.12_narration_alignment_s2500
model_size: 8B
num_battles: 10718
num_wins: 5012
ranking_group: single
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: rirv938/reward_gpt2_medium_preference_24m_e2
status: torndown
submission_type: basic
timestamp: 2024-08-14T03:38:51+00:00
us_pacific_date: 2024-08-13
win_ratio: 0.4676245568203023
Resubmit model
Running pipeline stage MKMLizer
Starting job with name bbchicago-brt-v1-12-narr-4893-v2-mkmlizer
Waiting for job on bbchicago-brt-v1-12-narr-4893-v2-mkmlizer to finish
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║     _____            __           __                                ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║       /___/                                                         ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║                                                                     ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  Version: 0.9.9                                                     ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  https://mk1.ai                                                     ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║                                                                     ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  belonging to:                                                      ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║                                                                     ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  Chai Research Corp.                                                ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ║                                                                     ║
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: Downloaded to shared memory in 46.148s
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpzyauyub3, device:0
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: quantized model in 30.563s
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: Processed model BBChicago/Brt_v1.12_narration_alignment_s2500 in 76.711s
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: creating bucket guanaco-mkml-models
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4893-v2
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4893-v2/config.json
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4893-v2/special_tokens_map.json
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4893-v2/tokenizer_config.json
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/bbchicago-brt-v1-12-narr-4893-v2/tokenizer.json
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: loading reward model from rirv938/reward_gpt2_medium_preference_24m_e2
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:11, 25.63it/s]
Loading 0:   4%|▍         | 12/291 [00:00<00:07, 35.53it/s]
Loading 0:   5%|▌         | 16/291 [00:00<00:08, 31.60it/s]
Loading 0:   7%|▋         | 21/291 [00:00<00:07, 34.46it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:08, 31.97it/s]
Loading 0:  10%|█         | 30/291 [00:00<00:07, 36.06it/s]
Loading 0:  12%|█▏        | 34/291 [00:01<00:10, 24.09it/s]
Loading 0:  13%|█▎        | 37/291 [00:01<00:10, 23.17it/s]
Loading 0:  14%|█▍        | 41/291 [00:01<00:10, 22.88it/s]
Loading 0:  16%|█▋        | 48/291 [00:01<00:07, 30.38it/s]
Loading 0:  18%|█▊        | 52/291 [00:01<00:08, 29.32it/s]
Loading 0:  20%|█▉        | 57/291 [00:01<00:07, 32.16it/s]
Loading 0:  21%|██        | 61/291 [00:02<00:07, 30.95it/s]
Loading 0:  23%|██▎       | 66/291 [00:02<00:06, 33.01it/s]
Loading 0:  24%|██▍       | 70/291 [00:02<00:06, 31.87it/s]
Loading 0:  25%|██▌       | 74/291 [00:02<00:06, 32.41it/s]
Loading 0:  27%|██▋       | 78/291 [00:02<00:06, 31.77it/s]
Loading 0:  28%|██▊       | 82/291 [00:02<00:09, 21.89it/s]
Loading 0:  29%|██▉       | 85/291 [00:03<00:08, 23.01it/s]
Loading 0:  31%|███       | 90/291 [00:03<00:07, 26.84it/s]
Loading 0:  32%|███▏      | 94/291 [00:03<00:07, 27.58it/s]
Loading 0:  34%|███▍      | 99/291 [00:03<00:06, 31.80it/s]
Loading 0:  35%|███▌      | 103/291 [00:03<00:06, 30.46it/s]
Loading 0:  37%|███▋      | 108/291 [00:03<00:05, 32.35it/s]
Loading 0:  38%|███▊      | 112/291 [00:03<00:05, 31.35it/s]
Loading 0:  40%|███▉      | 116/291 [00:03<00:05, 32.66it/s]
Loading 0:  42%|████▏     | 122/291 [00:04<00:04, 36.65it/s]
Loading 0:  44%|████▎     | 127/291 [00:04<00:04, 34.31it/s]
Loading 0:  46%|████▌     | 133/291 [00:04<00:05, 29.86it/s]
Loading 0:  47%|████▋     | 137/291 [00:04<00:05, 28.83it/s]
Loading 0:  48%|████▊     | 141/291 [00:04<00:05, 26.69it/s]
Loading 0:  50%|████▉     | 145/291 [00:04<00:04, 29.22it/s]
Loading 0:  51%|█████     | 149/291 [00:05<00:05, 26.79it/s]
Loading 0:  54%|█████▎    | 156/291 [00:05<00:03, 34.87it/s]
Loading 0:  55%|█████▍    | 160/291 [00:05<00:03, 33.92it/s]
Loading 0:  57%|█████▋    | 165/291 [00:05<00:03, 36.75it/s]
Loading 0:  58%|█████▊    | 169/291 [00:05<00:03, 35.16it/s]
Loading 0:  60%|█████▉    | 174/291 [00:05<00:03, 38.13it/s]
Loading 0:  61%|██████    | 178/291 [00:05<00:03, 36.16it/s]
Loading 0:  63%|██████▎   | 184/291 [00:05<00:02, 42.16it/s]
Loading 0:  65%|██████▍   | 189/291 [00:06<00:03, 25.62it/s]
Loading 0:  67%|██████▋   | 194/291 [00:06<00:03, 26.65it/s]
Loading 0:  69%|██████▉   | 201/291 [00:06<00:02, 33.03it/s]
Loading 0:  70%|███████   | 205/291 [00:06<00:02, 32.42it/s]
Loading 0:  72%|███████▏  | 210/291 [00:06<00:02, 32.87it/s]
Loading 0:  74%|███████▎  | 214/291 [00:07<00:03, 22.24it/s]
Loading 0:  75%|███████▌  | 219/291 [00:07<00:02, 25.33it/s]
Loading 0:  77%|███████▋  | 223/291 [00:07<00:02, 26.69it/s]
Loading 0:  78%|███████▊  | 227/291 [00:07<00:02, 28.85it/s]
Loading 0:  79%|███████▉  | 231/291 [00:07<00:01, 30.10it/s]
Loading 0:  81%|████████  | 235/291 [00:07<00:02, 23.67it/s]
Loading 0:  82%|████████▏ | 239/291 [00:08<00:02, 23.40it/s]
Loading 0:  84%|████████▍ | 244/291 [00:08<00:01, 28.45it/s]
Loading 0:  85%|████████▌ | 248/291 [00:08<00:01, 25.81it/s]
Loading 0:  88%|████████▊ | 255/291 [00:08<00:01, 32.07it/s]
Loading 0:  89%|████████▉ | 259/291 [00:08<00:01, 30.56it/s]
Loading 0:  91%|█████████ | 264/291 [00:08<00:00, 31.58it/s]
Loading 0:  92%|█████████▏| 268/291 [00:09<00:00, 29.98it/s]
Loading 0:  94%|█████████▍| 273/291 [00:09<00:00, 31.96it/s]
Loading 0:  95%|█████████▌| 277/291 [00:09<00:00, 30.73it/s]
Loading 0:  97%|█████████▋| 281/291 [00:09<00:00, 31.04it/s]
Loading 0:  98%|█████████▊| 286/291 [00:15<00:02,  2.45it/s]
Loading 0:  99%|█████████▉| 289/291 [00:15<00:00,  3.03it/s]
                                                            
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer:   warnings.warn(
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer:   warnings.warn(
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer:   warnings.warn(
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: Saving duration: 0.317s
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: Processed model rirv938/reward_gpt2_medium_preference_24m_e2 in 6.076s
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: creating bucket guanaco-reward-models
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: Bucket 's3://guanaco-reward-models/' created
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4893-v2_reward
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4893-v2_reward/config.json
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4893-v2_reward/tokenizer_config.json
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4893-v2_reward/special_tokens_map.json
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4893-v2_reward/merges.txt
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4893-v2_reward/vocab.json
bbchicago-brt-v1-12-narr-4893-v2-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/bbchicago-brt-v1-12-narr-4893-v2_reward/tokenizer.json
Job bbchicago-brt-v1-12-narr-4893-v2-mkmlizer completed after 124.79s with status: succeeded
Stopping job with name bbchicago-brt-v1-12-narr-4893-v2-mkmlizer
Pipeline stage MKMLizer completed in 126.47s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.09s
Running pipeline stage ISVCDeployer
Creating inference service bbchicago-brt-v1-12-narr-4893-v2
Waiting for inference service bbchicago-brt-v1-12-narr-4893-v2 to be ready
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nemo-9330-v47-mkmlizer
Waiting for job on mistralai-mistral-nemo-9330-v47-mkmlizer to finish
mistralai-mistral-nemo-9330-v47-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-nemo-9330-v47-mkmlizer: ║     _____            __           __                                ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║       /___/                                                         ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  Version: 0.9.9                                                     ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  belonging to:                                                      ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v47-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Inference service bbchicago-brt-v1-12-narr-4893-v2 ready after 300.9702067375183s
Pipeline stage ISVCDeployer completed in 303.14s
Running pipeline stage StressChecker
mistralai-mistral-nemo-9330-v47-mkmlizer: Downloaded to shared memory in 53.486s
mistralai-mistral-nemo-9330-v47-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpcruv1veu, device:0
mistralai-mistral-nemo-9330-v47-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Received healthy response to inference request in 2.0866687297821045s
Received healthy response to inference request in 1.1396889686584473s
Received healthy response to inference request in 1.1749725341796875s
Received healthy response to inference request in 1.142134666442871s
Received healthy response to inference request in 1.1460075378417969s
5 requests
0 failed requests
5th percentile: 1.140178108215332
10th percentile: 1.1406672477722168
20th percentile: 1.1416455268859864
30th percentile: 1.1429092407226562
40th percentile: 1.1444583892822267
50th percentile: 1.1460075378417969
60th percentile: 1.1575935363769532
70th percentile: 1.1691795349121095
80th percentile: 1.3573117733001712
90th percentile: 1.7219902515411378
95th percentile: 1.9043294906616208
99th percentile: 2.0502008819580078
mean time: 1.3378944873809815
Pipeline stage StressChecker completed in 8.10s
bbchicago-brt-v1-12-narr_4893_v2 status is now deployed due to DeploymentManager action
bbchicago-brt-v1-12-narr_4893_v2 status is now inactive due to auto deactivation removed underperforming models
bbchicago-brt-v1-12-narr_4893_v2 status is now torndown due to DeploymentManager action