submission_id: chaiml-sao10k-l3-rp-v3-3_v47
developer_uid: chai_backend_admin
alignment_samples: 0
best_of: 8
celo_rating: 1237.09
display_name: chaiml-sao10k-hollow
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '""', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.95, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|end_header_id|>,', '<|eot_id|>,', '\n\n{user_name}'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
is_internal_developer: True
language_model: ChaiML/sao10k-l3-rp-v3-3
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: ChaiML/sao10k-l3-rp-v3-3
model_name: chaiml-sao10k-hollow
model_num_parameters: 8030261248.0
model_repo: ChaiML/sao10k-l3-rp-v3-3
model_size: 8B
num_battles: 12547
num_wins: 6869
propriety_score: 0.6932880203908242
propriety_total_count: 1177.0
ranking_group: single
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
status: torndown
submission_type: basic
timestamp: 2024-07-23T03:43:41+00:00
us_pacific_date: 2024-07-22
win_ratio: 0.5474615445923329
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer
Waiting for job on chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer to finish
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ _____ __ __ ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ /___/ ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ Version: 0.9.5.post3 ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ https://mk1.ai ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ belonging to: ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ Chai Research Corp. ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ║ ║
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Downloaded to shared memory in 22.233s
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp7ysuo2z1, device:0
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 1%| | 2/291 [00:04<11:25, 2.37s/it] Loading 0: 5%|▌ | 15/291 [00:04<01:05, 4.19it/s] Loading 0: 11%|█ | 32/291 [00:04<00:24, 10.79it/s] Loading 0: 17%|█▋ | 49/291 [00:05<00:12, 19.30it/s] Loading 0: 22%|██▏ | 63/291 [00:05<00:09, 24.56it/s] Loading 0: 26%|██▋ | 77/291 [00:05<00:06, 34.04it/s] Loading 0: 32%|███▏ | 94/291 [00:05<00:04, 48.07it/s] Loading 0: 37%|███▋ | 107/291 [00:05<00:03, 58.68it/s] Loading 0: 42%|████▏ | 122/291 [00:05<00:02, 72.37it/s] Loading 0: 48%|████▊ | 139/291 [00:05<00:01, 89.07it/s] Loading 0: 53%|█████▎ | 154/291 [00:05<00:01, 100.35it/s] Loading 0: 58%|█████▊ | 169/291 [00:06<00:01, 79.27it/s] Loading 0: 64%|██████▎ | 185/291 [00:06<00:01, 93.17it/s] Loading 0: 69%|██████▉ | 202/291 [00:06<00:00, 108.06it/s] Loading 0: 74%|███████▍ | 216/291 [00:06<00:00, 114.16it/s] Loading 0: 79%|███████▉ | 230/291 [00:06<00:00, 119.40it/s] Loading 0: 85%|████████▍ | 247/291 [00:06<00:00, 130.29it/s] Loading 0: 90%|█████████ | 262/291 [00:06<00:00, 134.12it/s] Loading 0: 95%|█████████▌| 277/291 [00:07<00:00, 90.99it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: quantized model in 21.693s
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Processed model ChaiML/sao10k-l3-rp-v3-3 in 43.926s
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: creating bucket guanaco-mkml-models
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v47
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v47/special_tokens_map.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v47/config.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v47/tokenizer_config.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v47/tokenizer.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v47/flywheel_model.0.safetensors
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:950: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:778: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Downloading shards: 0%| | 0/2 [00:00<?, ?it/s] Downloading shards: 50%|█████ | 1/2 [00:05<00:05, 5.56s/it] Downloading shards: 100%|██████████| 2/2 [00:09<00:00, 4.45s/it] Downloading shards: 100%|██████████| 2/2 [00:09<00:00, 4.62s/it]
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 2.52it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 4.13it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.77it/s]
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Saving duration: 1.393s
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 13.933s
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: creating bucket guanaco-reward-models
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: Bucket 's3://guanaco-reward-models/' created
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v47_reward
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v47_reward/special_tokens_map.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v47_reward/config.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v47_reward/tokenizer_config.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v47_reward/merges.txt
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v47_reward/vocab.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v47_reward/tokenizer.json
chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/chaiml-sao10k-l3-rp-v3-3-v47_reward/reward.tensors
Job chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer completed after 94.78s with status: succeeded
Stopping job with name chaiml-sao10k-l3-rp-v3-3-v47-mkmlizer
Pipeline stage MKMLizer completed in 95.85s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.09s
Running pipeline stage ISVCDeployer
Creating inference service chaiml-sao10k-l3-rp-v3-3-v47
Waiting for inference service chaiml-sao10k-l3-rp-v3-3-v47 to be ready
Inference service chaiml-sao10k-l3-rp-v3-3-v47 ready after 40.42450976371765s
Pipeline stage ISVCDeployer completed in 42.19s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.151634693145752s
Received healthy response to inference request in 1.1585736274719238s
Received healthy response to inference request in 1.1781291961669922s
Received healthy response to inference request in 1.1347851753234863s
Received healthy response to inference request in 1.1501212120056152s
5 requests
0 failed requests
5th percentile: 1.137852382659912
10th percentile: 1.1409195899963378
20th percentile: 1.1470540046691895
30th percentile: 1.151811695098877
40th percentile: 1.1551926612854004
50th percentile: 1.1585736274719238
60th percentile: 1.1663958549499511
70th percentile: 1.1742180824279784
80th percentile: 1.3728302955627443
90th percentile: 1.7622324943542482
95th percentile: 1.9569335937499999
99th percentile: 2.1126944732666013
mean time: 1.3546487808227539
Pipeline stage StressChecker completed in 7.39s
chaiml-sao10k-l3-rp-v3-3_v47 status is now deployed due to DeploymentManager action
chaiml-sao10k-l3-rp-v3-3_v47 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-sao10k-l3-rp-v3-3_v47
Running pipeline stage ISVCDeleter
Checking if service chaiml-sao10k-l3-rp-v3-3-v47 is running
Tearing down inference service chaiml-sao10k-l3-rp-v3-3-v47
Service chaiml-sao10k-l3-rp-v3-3-v47 has been torndown
Pipeline stage ISVCDeleter completed in 5.45s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-sao10k-l3-rp-v3-3-v47/config.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47/tokenizer.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key chaiml-sao10k-l3-rp-v3-3-v47_reward/config.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47_reward/merges.txt from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47_reward/reward.tensors from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v47_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 5.74s
chaiml-sao10k-l3-rp-v3-3_v47 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics