chaiml-sao10k-l3-rp-v3-3

submission_id: chaiml-sao10k-l3-rp-v3-3_v45

developer_uid: chai_backend_admin

best_of: 8

celo_rating: 1232.58

display_name: chaiml-sao10k-hallow

family_friendly_score: 0.0

formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 0.95, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|end_header_id|>,', '<|eot_id|>,', '\n\n{user_name}'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64, 'reward_max_tokens': 256}

is_internal_developer: True

language_model: ChaiML/sao10k-l3-rp-v3-3

max_input_tokens: 1024

max_output_tokens: 64

model_architecture: LlamaForCausalLM

model_group: ChaiML/sao10k-l3-rp-v3-3

model_name: chaiml-sao10k-hallow

model_num_parameters: 8030261248.0

model_repo: ChaiML/sao10k-l3-rp-v3-3

model_size: 8B

num_battles: 11380

num_wins: 6138

ranking_group: single

reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}

reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634

status: torndown

submission_type: basic

timestamp: 2024-07-23T02:56:01+00:00

us_pacific_date: 2024-07-22

win_ratio: 0.5393673110720563

Download Preference Data

Resubmit model

Running pipeline stage MKMLizer
Starting job with name chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer
Waiting for job on chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer to finish
Failed to get response for submission blend_meral_2024-07-13: ('http://mistralai-mixtral-8x7b-3473-v86-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"TypeError : SamplingParameters.__init__() got an unexpected keyword argument \'reward_max_tokens\'"}')
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║     _____            __           __                                ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║       /___/                                                         ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║                                                                     ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  Version: 0.9.5.post3                                               ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║                                                                     ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  belonging to:                                                      ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║                                                                     ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ║                                                                     ║
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: Downloaded to shared memory in 21.425s
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpbrt1hw8f, device:0
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission blend_meral_2024-07-13: ('http://neversleep-noromaid-v0-8068-v133-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"TypeError : SamplingParameters.__init__() got an unexpected keyword argument \'reward_max_tokens\'"}')
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/291 [00:04<11:51,  2.46s/it]
Loading 0:   5%|▍         | 14/291 [00:05<01:13,  3.77it/s]
Loading 0:  10%|▉         | 29/291 [00:05<00:27,  9.41it/s]
Loading 0:  14%|█▍        | 41/291 [00:05<00:16, 15.16it/s]
Loading 0:  19%|█▉        | 55/291 [00:05<00:09, 23.74it/s]
Loading 0:  23%|██▎       | 67/291 [00:05<00:08, 26.66it/s]
Loading 0:  27%|██▋       | 78/291 [00:05<00:06, 34.35it/s]
Loading 0:  33%|███▎      | 95/291 [00:05<00:03, 49.53it/s]
Loading 0:  38%|███▊      | 110/291 [00:06<00:02, 63.78it/s]
Loading 0:  42%|████▏     | 123/291 [00:06<00:02, 72.36it/s]
Loading 0:  48%|████▊     | 139/291 [00:06<00:01, 88.79it/s]
Loading 0:  52%|█████▏    | 152/291 [00:06<00:01, 96.30it/s]
Loading 0:  57%|█████▋    | 166/291 [00:06<00:01, 71.59it/s]
Loading 0:  61%|██████    | 177/291 [00:06<00:01, 78.10it/s]
Loading 0:  67%|██████▋   | 194/291 [00:06<00:01, 93.77it/s]
Loading 0:  73%|███████▎  | 211/291 [00:06<00:00, 106.53it/s]
Loading 0:  77%|███████▋  | 224/291 [00:07<00:00, 107.98it/s]
Loading 0:  82%|████████▏ | 239/291 [00:07<00:00, 114.16it/s]
Loading 0:  88%|████████▊ | 256/291 [00:07<00:00, 123.13it/s]
Loading 0:  93%|█████████▎| 270/291 [00:07<00:00, 82.69it/s] 
Loading 0:  98%|█████████▊| 284/291 [00:07<00:00, 92.18it/s]
                                                            
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: quantized model in 23.189s
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: Processed model ChaiML/sao10k-l3-rp-v3-3 in 44.614s
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: creating bucket guanaco-mkml-models
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v45
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v45/config.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v45/special_tokens_map.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v45/tokenizer_config.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v45/tokenizer.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v45/flywheel_model.0.safetensors
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:950: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer:   warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:778: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer:   warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: 
Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading shards:  50%|█████     | 1/2 [00:05<00:05,  5.37s/it]
Downloading shards: 100%|██████████| 2/2 [00:08<00:00,  3.86s/it]
Downloading shards: 100%|██████████| 2/2 [00:08<00:00,  4.09s/it]
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.37it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.92it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.57it/s]
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: Saving duration: 1.364s
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 13.062s
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: creating bucket guanaco-reward-models
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: Bucket 's3://guanaco-reward-models/' created
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v45_reward
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v45_reward/config.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v45_reward/special_tokens_map.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v45_reward/tokenizer_config.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v45_reward/merges.txt
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v45_reward/vocab.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v45_reward/tokenizer.json
chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v45_reward/reward.tensors
Failed to get response for submission mistralai-mixtral-8x7b-_3473_v92: ('http://mistralai-mixtral-8x7b-3473-v92-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mixtral-8x7b-_3473_v92: ('http://mistralai-mixtral-8x7b-3473-v92-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Job chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer completed after 96.52s with status: succeeded
Stopping job with name chaiml-sao10k-l3-rp-v3-3-v45-mkmlizer
Pipeline stage MKMLizer completed in 97.83s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.10s
Running pipeline stage ISVCDeployer
Creating inference service chaiml-sao10k-l3-rp-v3-3-v45
Waiting for inference service chaiml-sao10k-l3-rp-v3-3-v45 to be ready
Failed to get response for submission mistralai-mixtral-8x7b-_3473_v92: ('http://mistralai-mixtral-8x7b-3473-v92-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission trace2333-duduk-llama3-v1_v3: ('http://trace2333-duduk-llama3-v1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"TypeError : SamplingParameters.__init__() got an unexpected keyword argument \'reward_max_tokens\'"}')
Failed to get response for submission blend_meral_2024-07-13: ('http://neversleep-noromaid-v0-8068-v133-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"TypeError : SamplingParameters.__init__() got an unexpected keyword argument \'reward_max_tokens\'"}')
Inference service chaiml-sao10k-l3-rp-v3-3-v45 ready after 50.98316669464111s
Pipeline stage ISVCDeployer completed in 53.08s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0867855548858643s
Received healthy response to inference request in 1.3052904605865479s
Received healthy response to inference request in 1.310713768005371s
Received healthy response to inference request in 1.3148410320281982s
Received healthy response to inference request in 1.2609667778015137s
5 requests
0 failed requests
5th percentile: 1.2698315143585206
10th percentile: 1.2786962509155273
20th percentile: 1.296425724029541
30th percentile: 1.3063751220703126
40th percentile: 1.3085444450378418
50th percentile: 1.310713768005371
60th percentile: 1.312364673614502
70th percentile: 1.3140155792236328
80th percentile: 1.4692299365997317
90th percentile: 1.7780077457427979
95th percentile: 1.9323966503143308
99th percentile: 2.0559077739715574
mean time: 1.455719518661499
Pipeline stage StressChecker completed in 7.93s
chaiml-sao10k-l3-rp-v3-3_v45 status is now deployed due to DeploymentManager action
chaiml-sao10k-l3-rp-v3-3_v45 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-sao10k-l3-rp-v3-3_v45
Running pipeline stage ISVCDeleter
Checking if service chaiml-sao10k-l3-rp-v3-3-v45 is running
Tearing down inference service chaiml-sao10k-l3-rp-v3-3-v45
Service chaiml-sao10k-l3-rp-v3-3-v45 has been torndown
Pipeline stage ISVCDeleter completed in 4.29s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-sao10k-l3-rp-v3-3-v45/config.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45/tokenizer.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key chaiml-sao10k-l3-rp-v3-3-v45_reward/config.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45_reward/merges.txt from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45_reward/reward.tensors from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v45_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 5.41s
chaiml-sao10k-l3-rp-v3-3_v45 status is now torndown due to DeploymentManager action