submission_id: fredithefish-goatrp2_v2
developer_uid: fredi
status: inactive
model_repo: Fredithefish/GoatRP2
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
generation_params: {'temperature': 0.99, 'top_p': 0.92, 'top_k': 55, 'presence_penalty': 0.9, 'frequency_penalty': 0.3, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': "### Instruction:\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}\n\n### Input:\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:'}
reward_formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:'}
timestamp: 2024-03-29T17:17:09+00:00
model_name: fredithefish-goatrp2_v2
model_eval_status: success
safety_score: 0.97
entertaining: 7.26
stay_in_character: 8.5
user_preference: 7.64
double_thumbs_up: 188
thumbs_up: 324
thumbs_down: 131
num_battles: 38782
num_wins: 19232
win_ratio: 0.49590015986798
celo_rating: 1154.37
Resubmit model
Running pipeline stage MKMLizer
Starting job with name fredithefish-goatrp2-v2-mkmlizer
Waiting for job on fredithefish-goatrp2-v2-mkmlizer to finish
fredithefish-goatrp2-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
fredithefish-goatrp2-v2-mkmlizer: ║ _____ __ __ ║
fredithefish-goatrp2-v2-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
fredithefish-goatrp2-v2-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
fredithefish-goatrp2-v2-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
fredithefish-goatrp2-v2-mkmlizer: ║ /___/ ║
fredithefish-goatrp2-v2-mkmlizer: ║ ║
fredithefish-goatrp2-v2-mkmlizer: ║ Version: 0.6.11 ║
fredithefish-goatrp2-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
fredithefish-goatrp2-v2-mkmlizer: ║ ║
fredithefish-goatrp2-v2-mkmlizer: ║ The license key for the current software has been verified as ║
fredithefish-goatrp2-v2-mkmlizer: ║ belonging to: ║
fredithefish-goatrp2-v2-mkmlizer: ║ ║
fredithefish-goatrp2-v2-mkmlizer: ║ Chai Research Corp. ║
fredithefish-goatrp2-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
fredithefish-goatrp2-v2-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
fredithefish-goatrp2-v2-mkmlizer: ║ ║
fredithefish-goatrp2-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
fredithefish-goatrp2-v2-mkmlizer: .gitattributes: 0%| | 0.00/1.52k [00:00<?, ?B/s] .gitattributes: 100%|██████████| 1.52k/1.52k [00:00<00:00, 7.97MB/s]
fredithefish-goatrp2-v2-mkmlizer: README.md: 0%| | 0.00/1.06k [00:00<?, ?B/s] README.md: 100%|██████████| 1.06k/1.06k [00:00<00:00, 7.07MB/s]
fredithefish-goatrp2-v2-mkmlizer: config.json: 0%| | 0.00/702 [00:00<?, ?B/s] config.json: 100%|██████████| 702/702 [00:00<00:00, 5.37MB/s]
fredithefish-goatrp2-v2-mkmlizer: mergekit_config.yml: 0%| | 0.00/405 [00:00<?, ?B/s] mergekit_config.yml: 100%|██████████| 405/405 [00:00<00:00, 3.57MB/s]
fredithefish-goatrp2-v2-mkmlizer: model-00002-of-00003.safetensors: 0%| | 0.00/9.96G [00:00<?, ?B/s] model-00002-of-00003.safetensors: 0%| | 10.5M/9.96G [00:00<02:40, 62.1MB/s] model-00002-of-00003.safetensors: 0%| | 31.5M/9.96G [00:00<01:35, 103MB/s] model-00002-of-00003.safetensors: 1%|▏ | 126M/9.96G [00:00<00:26, 375MB/s] model-00002-of-00003.safetensors: 2%|▏ | 231M/9.96G [00:00<00:16, 587MB/s] model-00002-of-00003.safetensors: 4%|▍ | 440M/9.96G [00:00<00:09, 1.05GB/s] model-00002-of-00003.safetensors: 6%|▌ | 619M/9.96G [00:00<00:07, 1.24GB/s] model-00002-of-00003.safetensors: 8%|▊ | 765M/9.96G [00:00<00:07, 1.31GB/s] model-00002-of-00003.safetensors: 11%|█ | 1.09G/9.96G [00:00<00:04, 1.86GB/s] model-00002-of-00003.safetensors: 13%|█▎ | 1.29G/9.96G [00:01<00:06, 1.35GB/s] model-00002-of-00003.safetensors: 15%|█▍ | 1.46G/9.96G [00:01<00:07, 1.15GB/s] model-00002-of-00003.safetensors: 16%|█▌ | 1.59G/9.96G [00:01<00:08, 1.03GB/s] model-00002-of-00003.safetensors: 18%|█▊ | 1.78G/9.96G [00:01<00:06, 1.18GB/s] model-00002-of-00003.safetensors: 20%|█▉ | 1.95G/9.96G [00:01<00:06, 1.26GB/s] model-00002-of-00003.safetensors: 21%|██ | 2.10G/9.96G [00:01<00:07, 1.06GB/s] model-00002-of-00003.safetensors: 22%|██▏ | 2.22G/9.96G [00:02<00:07, 1.02GB/s] model-00002-of-00003.safetensors: 23%|██▎ | 2.34G/9.96G [00:02<00:07, 970MB/s] model-00002-of-00003.safetensors: 25%|██▌ | 2.50G/9.96G [00:02<00:06, 1.10GB/s] model-00002-of-00003.safetensors: 27%|██▋ | 2.69G/9.96G [00:02<00:05, 1.31GB/s] model-00002-of-00003.safetensors: 29%|██▉ | 2.88G/9.96G [00:02<00:04, 1.44GB/s] model-00002-of-00003.safetensors: 31%|███ | 3.04G/9.96G [00:02<00:04, 1.39GB/s] model-00002-of-00003.safetensors: 33%|███▎ | 3.26G/9.96G [00:02<00:04, 1.56GB/s] model-00002-of-00003.safetensors: 35%|███▍ | 3.44G/9.96G [00:02<00:04, 1.61GB/s] model-00002-of-00003.safetensors: 37%|███▋ | 3.70G/9.96G [00:03<00:03, 1.87GB/s] model-00002-of-00003.safetensors: 39%|███▉ | 3.90G/9.96G [00:03<00:03, 1.84GB/s] model-00002-of-00003.safetensors: 41%|████ | 4.10G/9.96G [00:03<00:03, 1.84GB/s] model-00002-of-00003.safetensors: 44%|████▍ | 4.37G/9.96G [00:03<00:02, 2.01GB/s] model-00002-of-00003.safetensors: 46%|████▌ | 4.58G/9.96G [00:03<00:02, 2.03GB/s] model-00002-of-00003.safetensors: 48%|████▊ | 4.79G/9.96G [00:03<00:02, 2.02GB/s] model-00002-of-00003.safetensors: 52%|█████▏ | 5.15G/9.96G [00:03<00:01, 2.42GB/s] model-00002-of-00003.safetensors: 54%|█████▍ | 5.42G/9.96G [00:03<00:01, 2.50GB/s] model-00002-of-00003.safetensors: 57%|█████▋ | 5.67G/9.96G [00:03<00:01, 2.34GB/s] model-00002-of-00003.safetensors: 59%|█████▉ | 5.91G/9.96G [00:04<00:01, 2.23GB/s] model-00002-of-00003.safetensors: 62%|██████▏ | 6.14G/9.96G [00:04<00:01, 2.08GB/s] model-00002-of-00003.safetensors: 64%|██████▍ | 6.36G/9.96G [00:04<00:01, 1.95GB/s] model-00002-of-00003.safetensors: 67%|██████▋ | 6.66G/9.96G [00:04<00:01, 2.14GB/s] model-00002-of-00003.safetensors: 69%|██████▉ | 6.88G/9.96G [00:04<00:01, 1.93GB/s] model-00002-of-00003.safetensors: 71%|███████ | 7.08G/9.96G [00:04<00:01, 1.93GB/s] model-00002-of-00003.safetensors: 73%|███████▎ | 7.29G/9.96G [00:04<00:01, 1.88GB/s] model-00002-of-00003.safetensors: 75%|███████▌ | 7.50G/9.96G [00:04<00:01, 1.84GB/s] model-00002-of-00003.safetensors: 77%|███████▋ | 7.69G/9.96G [00:04<00:01, 1.75GB/s] model-00002-of-00003.safetensors: 79%|███████▉ | 7.87G/9.96G [00:05<00:01, 1.78GB/s] model-00002-of-00003.safetensors: 82%|████████▏ | 8.15G/9.96G [00:05<00:00, 2.04GB/s] model-00002-of-00003.safetensors: 84%|████████▍ | 8.40G/9.96G [00:05<00:00, 2.11GB/s] model-00002-of-00003.safetensors: 87%|████████▋ | 8.66G/9.96G [00:05<00:00, 2.23GB/s] model-00002-of-00003.safetensors: 89%|████████▉ | 8.89G/9.96G [00:05<00:00, 2.13GB/s] model-00002-of-00003.safetensors: 92%|█████████▏| 9.11G/9.96G [00:05<00:00, 2.13GB/s] model-00002-of-00003.safetensors: 94%|█████████▍| 9.38G/9.96G [00:05<00:00, 2.29GB/s] model-00002-of-00003.safetensors: 97%|█████████▋| 9.65G/9.96G [00:05<00:00, 2.35GB/s] model-00002-of-00003.safetensors: 99%|█████████▉| 9.89G/9.96G [00:05<00:00, 2.07GB/s] model-00002-of-00003.safetensors: 100%|█████████▉| 9.96G/9.96G [00:06<00:00, 1.55GB/s]
fredithefish-goatrp2-v2-mkmlizer: model-00003-of-00003.safetensors: 0%| | 0.00/6.16G [00:00<?, ?B/s] model-00003-of-00003.safetensors: 0%| | 10.5M/6.16G [00:00<01:32, 66.7MB/s] model-00003-of-00003.safetensors: 1%| | 41.9M/6.16G [00:00<00:37, 162MB/s] model-00003-of-00003.safetensors: 3%|▎ | 168M/6.16G [00:00<00:10, 569MB/s] model-00003-of-00003.safetensors: 5%|▌ | 315M/6.16G [00:00<00:07, 790MB/s] model-00003-of-00003.safetensors: 7%|▋ | 409M/6.16G [00:00<00:07, 810MB/s] model-00003-of-00003.safetensors: 8%|▊ | 503M/6.16G [00:00<00:07, 768MB/s] model-00003-of-00003.safetensors: 10%|█ | 640M/6.16G [00:00<00:06, 861MB/s] model-00003-of-00003.safetensors: 12%|█▏ | 734M/6.16G [00:01<00:06, 810MB/s] model-00003-of-00003.safetensors: 15%|█▌ | 933M/6.16G [00:01<00:05, 1.02GB/s] model-00003-of-00003.safetensors: 17%|█▋ | 1.04G/6.16G [00:01<00:05, 938MB/s] model-00003-of-00003.safetensors: 18%|█▊ | 1.13G/6.16G [00:01<00:05, 869MB/s] model-00003-of-00003.safetensors: 21%|██ | 1.27G/6.16G [00:01<00:04, 990MB/s] model-00003-of-00003.safetensors: 23%|██▎ | 1.41G/6.16G [00:01<00:04, 1.04GB/s] model-00003-of-00003.safetensors: 25%|██▍ | 1.52G/6.16G [00:01<00:05, 919MB/s] model-00003-of-00003.safetensors: 26%|██▋ | 1.63G/6.16G [00:01<00:04, 923MB/s] model-00003-of-00003.safetensors: 29%|██▉ | 1.81G/6.16G [00:02<00:03, 1.16GB/s] model-00003-of-00003.safetensors: 32%|███▏ | 2.00G/6.16G [00:02<00:03, 1.32GB/s] model-00003-of-00003.safetensors: 35%|███▌ | 2.16G/6.16G [00:02<00:02, 1.37GB/s] model-00003-of-00003.safetensors: 38%|███▊ | 2.34G/6.16G [00:02<00:02, 1.47GB/s] model-00003-of-00003.safetensors: 42%|████▏ | 2.58G/6.16G [00:02<00:02, 1.69GB/s] model-00003-of-00003.safetensors: 45%|████▍ | 2.76G/6.16G [00:02<00:02, 1.68GB/s] model-00003-of-00003.safetensors: 50%|████▉ | 3.05G/6.16G [00:02<00:01, 1.99GB/s] model-00003-of-00003.safetensors: 53%|█████▎ | 3.26G/6.16G [00:02<00:01, 1.87GB/s] model-00003-of-00003.safetensors: 56%|█████▌ | 3.46G/6.16G [00:02<00:01, 1.79GB/s] model-00003-of-00003.safetensors: 59%|█████▉ | 3.65G/6.16G [00:03<00:01, 1.62GB/s] model-00003-of-00003.safetensors: 62%|██████▏ | 3.85G/6.16G [00:03<00:01, 1.71GB/s] model-00003-of-00003.safetensors: 66%|██████▌ | 4.06G/6.16G [00:03<00:01, 1.78GB/s] model-00003-of-00003.safetensors: 69%|██████▉ | 4.25G/6.16G [00:03<00:01, 1.62GB/s] model-00003-of-00003.safetensors: 72%|███████▏ | 4.46G/6.16G [00:03<00:00, 1.72GB/s] model-00003-of-00003.safetensors: 75%|███████▌ | 4.63G/6.16G [00:03<00:00, 1.67GB/s] model-00003-of-00003.safetensors: 78%|███████▊ | 4.81G/6.16G [00:03<00:00, 1.61GB/s] model-00003-of-00003.safetensors: 82%|████████▏ | 5.03G/6.16G [00:03<00:00, 1.76GB/s] model-00003-of-00003.safetensors: 85%|████████▍ | 5.22G/6.16G [00:04<00:00, 1.62GB/s] model-00003-of-00003.safetensors: 89%|████████▉ | 5.50G/6.16G [00:04<00:00, 1.66GB/s] model-00003-of-00003.safetensors: 92%|█████████▏| 5.68G/6.16G [00:04<00:00, 1.49GB/s] model-00003-of-00003.safetensors: 95%|█████████▍| 5.85G/6.16G [00:04<00:00, 1.47GB/s] model-00003-of-00003.safetensors: 98%|█████████▊| 6.04G/6.16G [00:04<00:00, 1.57GB/s] model-00003-of-00003.safetensors: 100%|█████████▉| 6.16G/6.16G [00:04<00:00, 1.29GB/s]
fredithefish-goatrp2-v2-mkmlizer: model.safetensors.index.json: 0%| | 0.00/28.5k [00:00<?, ?B/s] model.safetensors.index.json: 100%|██████████| 28.5k/28.5k [00:00<00:00, 143MB/s]
fredithefish-goatrp2-v2-mkmlizer: special_tokens_map.json: 0%| | 0.00/552 [00:00<?, ?B/s] special_tokens_map.json: 100%|██████████| 552/552 [00:00<00:00, 8.64MB/s]
fredithefish-goatrp2-v2-mkmlizer: Profiling: 0%| | 0/363 [00:00<?, ?it/s] Profiling: 0%| | 1/363 [00:01<10:31, 1.74s/it] Profiling: 2%|▏ | 9/363 [00:01<00:54, 6.54it/s] Profiling: 5%|▍ | 17/363 [00:01<00:25, 13.56it/s] Profiling: 6%|▋ | 23/363 [00:02<00:17, 19.13it/s] Profiling: 9%|▉ | 32/363 [00:02<00:11, 28.57it/s] Profiling: 11%|█▏ | 41/363 [00:02<00:08, 38.77it/s] Profiling: 13%|█▎ | 49/363 [00:02<00:07, 44.66it/s] Profiling: 15%|█▌ | 56/363 [00:02<00:06, 46.93it/s] Profiling: 18%|█▊ | 67/363 [00:02<00:05, 58.42it/s] Profiling: 21%|██ | 75/363 [00:02<00:04, 62.87it/s] Profiling: 23%|██▎ | 83/363 [00:02<00:04, 65.40it/s] Profiling: 25%|██▌ | 91/363 [00:02<00:03, 69.13it/s] Profiling: 28%|██▊ | 100/363 [00:03<00:03, 73.62it/s] Profiling: 30%|██▉ | 108/363 [00:03<00:03, 68.10it/s] Profiling: 32%|███▏ | 116/363 [00:03<00:03, 70.17it/s] Profiling: 34%|███▍ | 124/363 [00:03<00:03, 72.02it/s] Profiling: 36%|███▋ | 132/363 [00:03<00:03, 72.70it/s] Profiling: 39%|███▊ | 140/363 [00:04<00:07, 30.88it/s] Profiling: 41%|████ | 149/363 [00:04<00:05, 38.10it/s] Profiling: 44%|████▎ | 158/363 [00:04<00:04, 46.17it/s] Profiling: 46%|████▌ | 167/363 [00:04<00:03, 53.85it/s] Profiling: 48%|████▊ | 176/363 [00:04<00:03, 61.24it/s] Profiling: 51%|█████ | 185/363 [00:04<00:02, 67.38it/s] Profiling: 53%|█████▎ | 194/363 [00:04<00:02, 72.26it/s] Profiling: 56%|█████▌ | 203/363 [00:04<00:02, 76.24it/s] Profiling: 58%|█████▊ | 212/363 [00:04<00:01, 79.05it/s] Profiling: 61%|██████ | 221/363 [00:05<00:01, 80.95it/s] Profiling: 63%|██████▎ | 230/363 [00:05<00:01, 81.90it/s] Profiling: 66%|██████▌ | 239/363 [00:05<00:01, 82.70it/s] Profiling: 68%|██████▊ | 248/363 [00:05<00:01, 83.71it/s] Profiling: 71%|███████▏ | 259/363 [00:05<00:01, 86.81it/s] Profiling: 74%|███████▍ | 268/363 [00:05<00:01, 83.12it/s] Profiling: 77%|███████▋ | 279/363 [00:07<00:06, 13.25it/s] Profiling: 79%|███████▉ | 288/363 [00:07<00:04, 17.42it/s] Profiling: 82%|████████▏ | 296/363 [00:08<00:03, 21.85it/s] Profiling: 84%|████████▍ | 305/363 [00:08<00:02, 28.09it/s] Profiling: 87%|████████▋ | 314/363 [00:08<00:01, 35.09it/s] Profiling: 89%|████████▊ | 322/363 [00:08<00:00, 41.03it/s] Profiling: 91%|█████████ | 330/363 [00:08<00:00, 46.83it/s] Profiling: 93%|█████████▎| 339/363 [00:08<00:00, 54.59it/s] Profiling: 96%|█████████▌| 348/363 [00:08<00:00, 61.72it/s] Profiling: 98%|█████████▊| 357/363 [00:08<00:00, 67.14it/s] Profiling: 100%|██████████| 363/363 [00:09<00:00, 39.73it/s]
Exception raised while processing tagging_function
fredithefish-goatrp2-v2-mkmlizer: quantized model in 28.408s
fredithefish-goatrp2-v2-mkmlizer: Processed model Fredithefish/GoatRP2 in 54.506s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
fredithefish-goatrp2-v2-mkmlizer: creating bucket guanaco-mkml-models
fredithefish-goatrp2-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
fredithefish-goatrp2-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/fredithefish-goatrp2-v2
fredithefish-goatrp2-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/fredithefish-goatrp2-v2/config.json
fredithefish-goatrp2-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/fredithefish-goatrp2-v2/tokenizer_config.json
fredithefish-goatrp2-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/fredithefish-goatrp2-v2/tokenizer.model
fredithefish-goatrp2-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/fredithefish-goatrp2-v2/special_tokens_map.json
fredithefish-goatrp2-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/fredithefish-goatrp2-v2/tokenizer.json
Job fredithefish-goatrp2-v2-mkmlizer completed after 106.81s with status: succeeded
Stopping job with name fredithefish-goatrp2-v2-mkmlizer
Exception raised while processing tagging_function
Pipeline stage MKMLizer completed in 114.31s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 31.23s
Running pipeline stage ISVCDeployer
Creating inference service fredithefish-goatrp2-v2
Waiting for inference service fredithefish-goatrp2-v2 to be ready
Inference service fredithefish-goatrp2-v2 ready after 62.73822498321533s
Pipeline stage ISVCDeployer completed in 94.49s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.989501953125s
Received healthy response to inference request in 1.7116446495056152s
Received healthy response to inference request in 1.7587165832519531s
Received healthy response to inference request in 1.6182496547698975s
Received healthy response to inference request in 1.674476146697998s
5 requests
0 failed requests
5th percentile: 1.6294949531555176
10th percentile: 1.6407402515411378
20th percentile: 1.6632308483123779
30th percentile: 1.6819098472595215
40th percentile: 1.6967772483825683
50th percentile: 1.7116446495056152
60th percentile: 1.7304734230041503
70th percentile: 1.7493021965026856
80th percentile: 2.004873657226563
90th percentile: 2.4971878051757814
95th percentile: 2.7433448791503903
99th percentile: 2.9402705383300782
mean time: 1.9505177974700927
Pipeline stage StressChecker completed in 12.59s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 1.99s
Running pipeline stage DaemonicSafetyScorer
Running M-Eval for topic stay_in_character
Pipeline stage DaemonicSafetyScorer completed in 0.66s
M-Eval Dataset for topic stay_in_character is loaded
fredithefish-goatrp2_v2 status is now deployed due to DeploymentManager action
fredithefish-goatrp2_v2 status is now inactive due to auto deactivation removed underperforming models

Usage Metrics

Latency Metrics