submission_id: chaoticneutrals-poppy-po_3816_v5
developer_uid: robert_irvine
status: inactive
model_repo: ChaoticNeutrals/Poppy_Porpoise-0.72-L3-8B
reward_repo: rirv938/gpt2_medium_pairwise_60m_step_468750
generation_params: {'temperature': 0.95, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
reward_formatter: {'memory_template': 'Memory: {memory}\n', 'prompt_template': '{prompt}\n', 'bot_template': 'Bot: {message}\n', 'user_template': 'User: {message}\n', 'response_template': 'Bot:', 'truncate_by_message': False}
timestamp: 2024-06-10T04:39:45+00:00
model_name: chaoticneutrals-poppy-po_3816_v5
model_eval_status: success
model_group: ChaoticNeutrals/Poppy_Po
num_battles: 22664
num_wins: 11812
celo_rating: 1196.73
safety_score: 0.94
propriety_score: 0.6880208333333333
propriety_total_count: 1920.0
submission_type: basic
model_architecture: LlamaForCausalLM
model_num_parameters: 8030261248.0
best_of: 8
max_input_tokens: 512
max_output_tokens: 64
display_name: chaoticneutrals-poppy-po_3816_v5
ineligible_reason: propriety_total_count < 5000
language_model: ChaoticNeutrals/Poppy_Porpoise-0.72-L3-8B
model_size: 8B
reward_model: rirv938/gpt2_medium_pairwise_60m_step_468750
us_pacific_date: 2024-06-09
win_ratio: 0.5211789622308507
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chaoticneutrals-poppy-po-3816-v5-mkmlizer
Waiting for job on chaoticneutrals-poppy-po-3816-v5-mkmlizer to finish
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ _____ __ __ ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ /___/ ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ Version: 0.8.14 ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ https://mk1.ai ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ The license key for the current software has been verified as ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ belonging to: ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ Chai Research Corp. ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ║ ║
chaoticneutrals-poppy-po-3816-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaoticneutrals-poppy-po-3816-v5-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
chaoticneutrals-poppy-po-3816-v5-mkmlizer: warnings.warn(warning_message, FutureWarning)
chaoticneutrals-poppy-po-3816-v5-mkmlizer: Downloaded to shared memory in 137.792s
chaoticneutrals-poppy-po-3816-v5-mkmlizer: quantizing model to /dev/shm/model_cache
chaoticneutrals-poppy-po-3816-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaoticneutrals-poppy-po-3816-v5-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 2%|▏ | 7/291 [00:00<00:04, 67.60it/s] Loading 0: 7%|▋ | 20/291 [00:00<00:02, 93.14it/s] Loading 0: 10%|█ | 30/291 [00:00<00:03, 86.20it/s] Loading 0: 14%|█▍ | 42/291 [00:00<00:04, 58.93it/s] Loading 0: 17%|█▋ | 50/291 [00:00<00:03, 63.73it/s] Loading 0: 22%|██▏ | 63/291 [00:00<00:02, 77.05it/s] Loading 0: 25%|██▌ | 73/291 [00:00<00:02, 81.17it/s] Loading 0: 28%|██▊ | 82/291 [00:01<00:03, 59.39it/s] Loading 0: 32%|███▏ | 92/291 [00:01<00:03, 66.31it/s] Loading 0: 36%|███▌ | 105/291 [00:01<00:02, 80.66it/s] Loading 0: 40%|████ | 117/291 [00:01<00:02, 85.45it/s] Loading 0: 42%|████▏ | 123/291 [00:12<00:01, 85.45it/s] Loading 0: 43%|████▎ | 124/291 [00:12<00:58, 2.85it/s] Loading 0: 45%|████▌ | 132/291 [00:12<00:41, 3.80it/s] Loading 0: 49%|████▉ | 144/291 [00:13<00:26, 5.63it/s] Loading 0: 53%|█████▎ | 153/291 [00:13<00:18, 7.59it/s] Loading 0: 57%|█████▋ | 165/291 [00:13<00:11, 11.18it/s] Loading 0: 60%|██████ | 175/291 [00:13<00:07, 15.03it/s] Loading 0: 63%|██████▎ | 184/291 [00:13<00:06, 17.64it/s] Loading 0: 67%|██████▋ | 194/291 [00:13<00:04, 23.31it/s] Loading 0: 70%|███████ | 204/291 [00:13<00:02, 30.01it/s] Loading 0: 75%|███████▍ | 218/291 [00:14<00:01, 41.73it/s] Loading 0: 78%|███████▊ | 228/291 [00:14<00:01, 38.81it/s] Loading 0: 83%|████████▎ | 241/291 [00:14<00:01, 49.63it/s] Loading 0: 87%|████████▋ | 254/291 [00:14<00:00, 62.04it/s] Loading 0: 91%|█████████ | 264/291 [00:14<00:00, 57.58it/s] Loading 0: 94%|█████████▍| 274/291 [00:14<00:00, 64.46it/s] Loading 0: 98%|█████████▊| 284/291 [00:15<00:00, 71.38it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
chaoticneutrals-poppy-po-3816-v5-mkmlizer: quantized model in 26.611s
chaoticneutrals-poppy-po-3816-v5-mkmlizer: Processed model ChaoticNeutrals/Poppy_Porpoise-0.72-L3-8B in 167.338s
chaoticneutrals-poppy-po-3816-v5-mkmlizer: creating bucket guanaco-mkml-models
chaoticneutrals-poppy-po-3816-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaoticneutrals-poppy-po-3816-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaoticneutrals-poppy-po-3816-v5
chaoticneutrals-poppy-po-3816-v5-mkmlizer: DEBUG retryable error: RequestError: send request failed
chaoticneutrals-poppy-po-3816-v5-mkmlizer: caused by: dial tcp 216.153.53.63:443: i/o timeout
chaoticneutrals-poppy-po-3816-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaoticneutrals-poppy-po-3816-v5/special_tokens_map.json
chaoticneutrals-poppy-po-3816-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaoticneutrals-poppy-po-3816-v5/tokenizer_config.json
chaoticneutrals-poppy-po-3816-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaoticneutrals-poppy-po-3816-v5/config.json
chaoticneutrals-poppy-po-3816-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaoticneutrals-poppy-po-3816-v5/tokenizer.json
chaoticneutrals-poppy-po-3816-v5-mkmlizer: loading reward model from rirv938/gpt2_medium_pairwise_60m_step_468750
chaoticneutrals-poppy-po-3816-v5-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:913: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaoticneutrals-poppy-po-3816-v5-mkmlizer: warnings.warn(
chaoticneutrals-poppy-po-3816-v5-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:757: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaoticneutrals-poppy-po-3816-v5-mkmlizer: warnings.warn(
chaoticneutrals-poppy-po-3816-v5-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaoticneutrals-poppy-po-3816-v5-mkmlizer: warnings.warn(
chaoticneutrals-poppy-po-3816-v5-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
chaoticneutrals-poppy-po-3816-v5-mkmlizer: Saving duration: 0.468s
chaoticneutrals-poppy-po-3816-v5-mkmlizer: Processed model rirv938/gpt2_medium_pairwise_60m_step_468750 in 12.597s
chaoticneutrals-poppy-po-3816-v5-mkmlizer: creating bucket guanaco-reward-models
chaoticneutrals-poppy-po-3816-v5-mkmlizer: WARNING: Retrying failed request: /?location ([Errno 110] Connection timed out)
chaoticneutrals-poppy-po-3816-v5-mkmlizer: WARNING: Waiting 3 sec...
chaoticneutrals-poppy-po-3816-v5-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/chaoticneutrals-poppy-po-3816-v5_reward/reward.tensors
Job chaoticneutrals-poppy-po-3816-v5-mkmlizer completed after 369.1s with status: succeeded
Stopping job with name chaoticneutrals-poppy-po-3816-v5-mkmlizer
Pipeline stage MKMLizer completed in 371.66s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.09s
Running pipeline stage ISVCDeployer
Creating inference service chaoticneutrals-poppy-po-3816-v5
Waiting for inference service chaoticneutrals-poppy-po-3816-v5 to be ready
Inference service chaoticneutrals-poppy-po-3816-v5 ready after 311.58355689048767s
Pipeline stage ISVCDeployer completed in 318.54s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9798502922058105s
%s, retrying in %s seconds...
Received healthy response to inference request in 1.2226605415344238s
Received healthy response to inference request in 1.1819806098937988s
Received healthy response to inference request in 1.171074628829956s
Received healthy response to inference request in 1.173403024673462s
Received healthy response to inference request in 1.184279203414917s
5 requests
0 failed requests
5th percentile: 1.1715403079986573
10th percentile: 1.1720059871673585
20th percentile: 1.1729373455047607
30th percentile: 1.1751185417175294
40th percentile: 1.178549575805664
50th percentile: 1.1819806098937988
60th percentile: 1.1829000473022462
70th percentile: 1.1838194847106933
80th percentile: 1.1919554710388183
90th percentile: 1.207308006286621
95th percentile: 1.2149842739105225
99th percentile: 1.2211252880096435
mean time: 1.1866796016693115
Pipeline stage StressChecker completed in 28.61s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 0.03s
Running pipeline stage DaemonicSafetyScorer
Running M-Eval for topic stay_in_character
Pipeline stage DaemonicSafetyScorer completed in 0.04s
M-Eval Dataset for topic stay_in_character is loaded
chaoticneutrals-poppy-po_3816_v5 status is now deployed due to DeploymentManager action
chaoticneutrals-poppy-po_3816_v5 status is now inactive due to auto deactivation removed underperforming models

Usage Metrics

Latency Metrics