submission_id: chaiml-sao10k-l3-rp-v3-3_v6
developer_uid: chai_backend_admin
best_of: 8
celo_rating: 1156.87
display_name: chaiml-sao10k-l3-rp-v3-3_v6
family_friendly_score: 0.0
formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.95, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|end_header_id|>,', '<|eot_id|>,', '\n\n{user_name}'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
is_internal_developer: True
language_model: ChaiML/sao10k-l3-rp-v3-3
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: ChaiML/sao10k-l3-rp-v3-3
model_name: chaiml-sao10k-l3-rp-v3-3_v6
model_num_parameters: 8030261248.0
model_repo: ChaiML/sao10k-l3-rp-v3-3
model_size: 8B
num_battles: 32228
num_wins: 14857
ranking_group: single
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: Jellywibble/CHAI_alignment_reward_model
status: torndown
submission_type: basic
timestamp: 2024-07-06T00:38:01+00:00
us_pacific_date: 2024-07-05
win_ratio: 0.4609966488767531
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer
Waiting for job on chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer to finish
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║     _____            __           __                                ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║       /___/                                                         ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║                                                                     ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  Version: 0.8.14                                                    ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║                                                                     ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  belonging to:                                                      ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║                                                                     ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║                                                                     ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Downloaded to shared memory in 25.343s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: quantizing model to /dev/shm/model_cache
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/291 [00:04<11:20,  2.35s/it]
Loading 0:   4%|▍         | 13/291 [00:04<01:16,  3.65it/s]
Loading 0:   8%|▊         | 23/291 [00:04<00:35,  7.53it/s]
Loading 0:  11%|█▏        | 33/291 [00:05<00:20, 12.54it/s]
Loading 0:  17%|█▋        | 49/291 [00:05<00:10, 22.87it/s]
Loading 0:  21%|██        | 60/291 [00:05<00:09, 23.83it/s]
Loading 0:  26%|██▌       | 76/291 [00:05<00:06, 35.09it/s]
Loading 0:  30%|██▉       | 87/291 [00:05<00:04, 42.70it/s]
Loading 0:  35%|███▌      | 103/291 [00:05<00:03, 57.79it/s]
Loading 0:  39%|███▉      | 114/291 [00:06<00:02, 64.41it/s]
Loading 0:  45%|████▍     | 130/291 [00:06<00:02, 79.96it/s]
Loading 0:  49%|████▉     | 142/291 [00:06<00:01, 84.40it/s]
Loading 0:  54%|█████▍    | 157/291 [00:06<00:01, 95.79it/s]
Loading 0:  58%|█████▊    | 169/291 [00:06<00:01, 62.43it/s]
Loading 0:  63%|██████▎   | 184/291 [00:06<00:01, 76.39it/s]
Loading 0:  67%|██████▋   | 195/291 [00:06<00:01, 80.16it/s]
Loading 0:  73%|███████▎  | 211/291 [00:07<00:00, 95.11it/s]
Loading 0:  77%|███████▋  | 223/291 [00:07<00:00, 99.36it/s]
Loading 0:  82%|████████▏ | 239/291 [00:07<00:00, 109.75it/s]
Loading 0:  87%|████████▋ | 253/291 [00:07<00:00, 116.55it/s]
Loading 0:  91%|█████████▏| 266/291 [00:07<00:00, 70.80it/s] 
Loading 0:  95%|█████████▍| 276/291 [00:07<00:00, 75.50it/s]
                                                            
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: quantized model in 24.186s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Processed model ChaiML/sao10k-l3-rp-v3-3 in 49.530s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: creating bucket guanaco-mkml-models
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/config.json
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/special_tokens_map.json
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/tokenizer_config.json
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/tokenizer.json
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/flywheel_model.0.safetensors
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: loading reward model from Jellywibble/CHAI_alignment_reward_model
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:919: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer:   warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer:   warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:769: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer:   warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer:   warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Saving duration: 0.150s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Processed model Jellywibble/CHAI_alignment_reward_model in 6.222s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: creating bucket guanaco-reward-models
Job chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer completed after 134.07s with status: succeeded
Stopping job with name chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer
Pipeline stage MKMLizer completed in 136.11s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.42s
Running pipeline stage ISVCDeployer
Creating inference service chaiml-sao10k-l3-rp-v3-3-v6
Waiting for inference service chaiml-sao10k-l3-rp-v3-3-v6 to be ready
Inference service chaiml-sao10k-l3-rp-v3-3-v6 ready after 51.05707311630249s
Pipeline stage ISVCDeployer completed in 53.34s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.3435118198394775s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.618872880935669s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.6351609230041504s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.694105863571167s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.5980210304260254s
5 requests
0 failed requests
5th percentile: 1.602191400527954
10th percentile: 1.6063617706298827
20th percentile: 1.6147025108337403
30th percentile: 1.6221304893493653
40th percentile: 1.6286457061767579
50th percentile: 1.6351609230041504
60th percentile: 1.658738899230957
70th percentile: 1.6823168754577638
80th percentile: 1.8239870548248291
90th percentile: 2.0837494373321532
95th percentile: 2.213630628585815
99th percentile: 2.317535581588745
mean time: 1.7779345035552978
Pipeline stage StressChecker completed in 12.72s
chaiml-sao10k-l3-rp-v3-3_v6 status is now deployed due to DeploymentManager action
chaiml-sao10k-l3-rp-v3-3_v6 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-sao10k-l3-rp-v3-3_v6
Running pipeline stage ISVCDeleter
Checking if service chaiml-sao10k-l3-rp-v3-3-v6 is running
Tearing down inference service chaiml-sao10k-l3-rp-v3-3-v6
Service chaiml-sao10k-l3-rp-v3-3-v6 has been torndown
Pipeline stage ISVCDeleter completed in 6.27s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/config.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/tokenizer.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/config.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/merges.txt from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/reward.tensors from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 5.51s
chaiml-sao10k-l3-rp-v3-3_v6 status is now torndown due to DeploymentManager action