submission_id: fredithefish-starfishrp_v4
developer_uid: fredi
status: inactive
model_repo: Fredithefish/StarfishRP
reward_repo: Fredithefish/crazy_reward
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:'}
reward_formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:'}
timestamp: 2024-03-28T21:54:10+00:00
model_name: fredithefish-starfishrp_v4
model_eval_status: success
safety_score: 0.99
entertaining: 6.96
stay_in_character: 8.54
user_preference: 7.64
double_thumbs_up: 235
thumbs_up: 354
thumbs_down: 159
num_battles: 50151
num_wins: 22919
win_ratio: 0.456999860421527
celo_rating: 1127.56
Resubmit model
Running pipeline stage MKMLizer
Starting job with name fredithefish-starfishrp-v4-mkmlizer
Waiting for job on fredithefish-starfishrp-v4-mkmlizer to finish
fredithefish-starfishrp-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
fredithefish-starfishrp-v4-mkmlizer: ║ _____ __ __ ║
fredithefish-starfishrp-v4-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
fredithefish-starfishrp-v4-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
fredithefish-starfishrp-v4-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
fredithefish-starfishrp-v4-mkmlizer: ║ /___/ ║
fredithefish-starfishrp-v4-mkmlizer: ║ ║
fredithefish-starfishrp-v4-mkmlizer: ║ Version: 0.6.11 ║
fredithefish-starfishrp-v4-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
fredithefish-starfishrp-v4-mkmlizer: ║ ║
fredithefish-starfishrp-v4-mkmlizer: ║ The license key for the current software has been verified as ║
fredithefish-starfishrp-v4-mkmlizer: ║ belonging to: ║
fredithefish-starfishrp-v4-mkmlizer: ║ ║
fredithefish-starfishrp-v4-mkmlizer: ║ Chai Research Corp. ║
fredithefish-starfishrp-v4-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
fredithefish-starfishrp-v4-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
fredithefish-starfishrp-v4-mkmlizer: ║ ║
fredithefish-starfishrp-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
fredithefish-starfishrp-v4-mkmlizer: .gitattributes: 0%| | 0.00/1.52k [00:00<?, ?B/s] .gitattributes: 100%|██████████| 1.52k/1.52k [00:00<00:00, 16.4MB/s]
fredithefish-starfishrp-v4-mkmlizer: README.md: 0%| | 0.00/470 [00:00<?, ?B/s] README.md: 100%|██████████| 470/470 [00:00<00:00, 6.89MB/s]
fredithefish-starfishrp-v4-mkmlizer: config.json: 0%| | 0.00/641 [00:00<?, ?B/s] config.json: 100%|██████████| 641/641 [00:00<00:00, 6.97MB/s]
fredithefish-starfishrp-v4-mkmlizer: mergekit_config.yml: 0%| | 0.00/678 [00:00<?, ?B/s] mergekit_config.yml: 100%|██████████| 678/678 [00:00<00:00, 8.34MB/s]
fredithefish-starfishrp-v4-mkmlizer: model-00002-of-00002.safetensors: 0%| | 0.00/4.54G [00:00<?, ?B/s] model-00002-of-00002.safetensors: 0%| | 21.0M/4.54G [00:00<00:22, 198MB/s] model-00002-of-00002.safetensors: 3%|▎ | 136M/4.54G [00:00<00:06, 671MB/s] model-00002-of-00002.safetensors: 6%|▌ | 252M/4.54G [00:00<00:05, 847MB/s] model-00002-of-00002.safetensors: 10%|▉ | 451M/4.54G [00:00<00:03, 1.27GB/s] model-00002-of-00002.safetensors: 15%|█▌ | 692M/4.54G [00:00<00:02, 1.66GB/s] model-00002-of-00002.safetensors: 21%|██ | 944M/4.54G [00:00<00:01, 1.90GB/s] model-00002-of-00002.safetensors: 25%|██▌ | 1.14G/4.54G [00:00<00:01, 1.85GB/s] model-00002-of-00002.safetensors: 29%|██▉ | 1.33G/4.54G [00:00<00:01, 1.72GB/s] model-00002-of-00002.safetensors: 33%|███▎ | 1.51G/4.54G [00:00<00:01, 1.68GB/s] model-00002-of-00002.safetensors: 37%|███▋ | 1.69G/4.54G [00:01<00:01, 1.68GB/s] model-00002-of-00002.safetensors: 41%|████ | 1.87G/4.54G [00:01<00:01, 1.47GB/s] model-00002-of-00002.safetensors: 45%|████▍ | 2.03G/4.54G [00:01<00:01, 1.51GB/s] model-00002-of-00002.safetensors: 48%|████▊ | 2.19G/4.54G [00:01<00:01, 1.42GB/s] model-00002-of-00002.safetensors: 51%|█████▏ | 2.34G/4.54G [00:01<00:01, 1.34GB/s] model-00002-of-00002.safetensors: 55%|█████▍ | 2.49G/4.54G [00:01<00:01, 1.37GB/s] model-00002-of-00002.safetensors: 58%|█████▊ | 2.64G/4.54G [00:01<00:01, 1.39GB/s] model-00002-of-00002.safetensors: 61%|██████▏ | 2.79G/4.54G [00:01<00:01, 1.21GB/s] model-00002-of-00002.safetensors: 66%|██████▋ | 3.01G/4.54G [00:02<00:01, 1.43GB/s] model-00002-of-00002.safetensors: 70%|██████▉ | 3.17G/4.54G [00:02<00:01, 1.34GB/s] model-00002-of-00002.safetensors: 76%|███████▋ | 3.47G/4.54G [00:02<00:00, 1.76GB/s] model-00002-of-00002.safetensors: 81%|████████ | 3.66G/4.54G [00:02<00:00, 1.75GB/s] model-00002-of-00002.safetensors: 87%|████████▋ | 3.96G/4.54G [00:02<00:00, 2.08GB/s] model-00002-of-00002.safetensors: 97%|█████████▋| 4.40G/4.54G [00:02<00:00, 2.71GB/s] model-00002-of-00002.safetensors: 100%|█████████▉| 4.54G/4.54G [00:02<00:00, 1.68GB/s]
fredithefish-starfishrp-v4-mkmlizer: model.safetensors.index.json: 0%| | 0.00/22.8k [00:00<?, ?B/s] model.safetensors.index.json: 100%|██████████| 22.8k/22.8k [00:00<00:00, 98.1MB/s]
fredithefish-starfishrp-v4-mkmlizer: special_tokens_map.json: 0%| | 0.00/414 [00:00<?, ?B/s] special_tokens_map.json: 100%|██████████| 414/414 [00:00<00:00, 4.59MB/s]
fredithefish-starfishrp-v4-mkmlizer: tokenizer.json: 0%| | 0.00/1.80M [00:00<?, ?B/s] tokenizer.json: 100%|██████████| 1.80M/1.80M [00:00<00:00, 15.2MB/s] tokenizer.json: 100%|██████████| 1.80M/1.80M [00:00<00:00, 15.1MB/s]
fredithefish-starfishrp-v4-mkmlizer: tokenizer.model: 0%| | 0.00/493k [00:00<?, ?B/s] tokenizer.model: 100%|██████████| 493k/493k [00:00<00:00, 58.1MB/s]
fredithefish-starfishrp-v4-mkmlizer: tokenizer_config.json: 0%| | 0.00/967 [00:00<?, ?B/s] tokenizer_config.json: 100%|██████████| 967/967 [00:00<00:00, 7.32MB/s]
fredithefish-starfishrp-v4-mkmlizer: Downloaded to shared memory in 51.457s
fredithefish-starfishrp-v4-mkmlizer: quantizing model to /dev/shm/model_cache
fredithefish-starfishrp-v4-mkmlizer: Saving mkml model at /dev/shm/model_cache
fredithefish-starfishrp-v4-mkmlizer: Reading /tmp/tmp1xuh9u4j/model.safetensors.index.json
fredithefish-starfishrp-v4-mkmlizer: Profiling: 0%| | 0/291 [00:00<?, ?it/s] Profiling: 0%| | 1/291 [00:01<06:39, 1.38s/it] Profiling: 4%|▍ | 12/291 [00:01<00:25, 10.91it/s] Profiling: 8%|▊ | 23/291 [00:01<00:12, 22.04it/s] Profiling: 13%|█▎ | 39/291 [00:01<00:06, 40.91it/s] Profiling: 17%|█▋ | 50/291 [00:01<00:04, 51.18it/s] Profiling: 23%|██▎ | 66/291 [00:01<00:03, 70.58it/s] Profiling: 27%|██▋ | 78/291 [00:02<00:02, 78.29it/s] Profiling: 32%|███▏ | 93/291 [00:02<00:02, 92.75it/s] Profiling: 36%|███▋ | 106/291 [00:02<00:01, 97.34it/s] Profiling: 41%|████ | 120/291 [00:02<00:01, 105.95it/s] Profiling: 46%|████▌ | 133/291 [00:02<00:01, 106.92it/s] Profiling: 51%|█████ | 147/291 [00:02<00:01, 114.50it/s] Profiling: 55%|█████▍ | 160/291 [00:02<00:01, 117.79it/s] Profiling: 59%|█████▉ | 173/291 [00:02<00:01, 114.92it/s] Profiling: 64%|██████▍ | 187/291 [00:02<00:00, 117.05it/s] Profiling: 69%|██████▊ | 200/291 [00:03<00:00, 107.07it/s] Profiling: 73%|███████▎ | 212/291 [00:05<00:04, 17.42it/s] Profiling: 77%|███████▋ | 223/291 [00:05<00:03, 22.48it/s] Profiling: 82%|████████▏ | 238/291 [00:05<00:01, 31.47it/s] Profiling: 86%|████████▌ | 249/291 [00:05<00:01, 37.91it/s] Profiling: 91%|█████████ | 265/291 [00:05<00:00, 51.30it/s] Profiling: 95%|█████████▌| 277/291 [00:05<00:00, 59.42it/s] Profiling: 100%|█████████▉| 290/291 [00:05<00:00, 70.90it/s] Profiling: 100%|██████████| 291/291 [00:06<00:00, 47.00it/s]
fredithefish-starfishrp-v4-mkmlizer: quantized model in 17.458s
fredithefish-starfishrp-v4-mkmlizer: Processed model Fredithefish/StarfishRP in 70.214s
fredithefish-starfishrp-v4-mkmlizer: creating bucket guanaco-mkml-models
fredithefish-starfishrp-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
fredithefish-starfishrp-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/fredithefish-starfishrp-v4
fredithefish-starfishrp-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/fredithefish-starfishrp-v4/config.json
fredithefish-starfishrp-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/fredithefish-starfishrp-v4/special_tokens_map.json
fredithefish-starfishrp-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/fredithefish-starfishrp-v4/tokenizer_config.json
fredithefish-starfishrp-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/fredithefish-starfishrp-v4/tokenizer.json
fredithefish-starfishrp-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/fredithefish-starfishrp-v4/tokenizer.model
fredithefish-starfishrp-v4-mkmlizer: cp /dev/shm/model_cache/mkml_model.tensors s3://guanaco-mkml-models/fredithefish-starfishrp-v4/mkml_model.tensors
fredithefish-starfishrp-v4-mkmlizer: loading reward model from Fredithefish/crazy_reward
fredithefish-starfishrp-v4-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1067: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
fredithefish-starfishrp-v4-mkmlizer: warnings.warn(
fredithefish-starfishrp-v4-mkmlizer: config.json: 0%| | 0.00/1.02k [00:00<?, ?B/s] config.json: 100%|██████████| 1.02k/1.02k [00:00<00:00, 9.50MB/s]
fredithefish-starfishrp-v4-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:690: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
fredithefish-starfishrp-v4-mkmlizer: warnings.warn(
fredithefish-starfishrp-v4-mkmlizer: tokenizer_config.json: 0%| | 0.00/477 [00:00<?, ?B/s] tokenizer_config.json: 100%|██████████| 477/477 [00:00<00:00, 4.92MB/s]
fredithefish-starfishrp-v4-mkmlizer: vocab.json: 0%| | 0.00/798k [00:00<?, ?B/s] vocab.json: 100%|██████████| 798k/798k [00:00<00:00, 8.74MB/s]
fredithefish-starfishrp-v4-mkmlizer: merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s] merges.txt: 100%|██████████| 456k/456k [00:00<00:00, 18.6MB/s]
fredithefish-starfishrp-v4-mkmlizer: tokenizer.json: 0%| | 0.00/2.11M [00:00<?, ?B/s] tokenizer.json: 100%|██████████| 2.11M/2.11M [00:00<00:00, 32.9MB/s]
fredithefish-starfishrp-v4-mkmlizer: special_tokens_map.json: 0%| | 0.00/587 [00:00<?, ?B/s] special_tokens_map.json: 100%|██████████| 587/587 [00:00<00:00, 8.67MB/s]
fredithefish-starfishrp-v4-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
fredithefish-starfishrp-v4-mkmlizer: warnings.warn(
fredithefish-starfishrp-v4-mkmlizer: model.safetensors.index.json: 0%| | 0.00/10.5k [00:00<?, ?B/s] model.safetensors.index.json: 100%|██████████| 10.5k/10.5k [00:00<00:00, 54.2MB/s]
fredithefish-starfishrp-v4-mkmlizer: Downloading shards: 0%| | 0/1 [00:00<?, ?it/s]
fredithefish-starfishrp-v4-mkmlizer: model-00001-of-00001.safetensors: 0%| | 0.00/249M [00:00<?, ?B/s]
fredithefish-starfishrp-v4-mkmlizer: model-00001-of-00001.safetensors: 4%|▍ | 10.5M/249M [00:00<00:02, 95.2MB/s]
fredithefish-starfishrp-v4-mkmlizer: model-00001-of-00001.safetensors: 8%|▊ | 21.0M/249M [00:00<00:02, 98.8MB/s]
fredithefish-starfishrp-v4-mkmlizer: model-00001-of-00001.safetensors: 66%|██████▋ | 165M/249M [00:00<00:00, 683MB/s] 
fredithefish-starfishrp-v4-mkmlizer: model-00001-of-00001.safetensors: 96%|█████████▌| 238M/249M [00:00<00:00, 585MB/s] model-00001-of-00001.safetensors: 100%|█████████▉| 249M/249M [00:00<00:00, 462MB/s]
fredithefish-starfishrp-v4-mkmlizer: Downloading shards: 100%|██████████| 1/1 [00:00<00:00, 1.35it/s] Downloading shards: 100%|██████████| 1/1 [00:00<00:00, 1.35it/s]
fredithefish-starfishrp-v4-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
fredithefish-starfishrp-v4-mkmlizer: Saving duration: 0.095s
fredithefish-starfishrp-v4-mkmlizer: Processed model Fredithefish/crazy_reward in 2.943s
fredithefish-starfishrp-v4-mkmlizer: creating bucket guanaco-reward-models
fredithefish-starfishrp-v4-mkmlizer: Bucket 's3://guanaco-reward-models/' created
fredithefish-starfishrp-v4-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/fredithefish-starfishrp-v4_reward
fredithefish-starfishrp-v4-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/fredithefish-starfishrp-v4_reward/config.json
fredithefish-starfishrp-v4-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/fredithefish-starfishrp-v4_reward/tokenizer_config.json
fredithefish-starfishrp-v4-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/fredithefish-starfishrp-v4_reward/special_tokens_map.json
fredithefish-starfishrp-v4-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/fredithefish-starfishrp-v4_reward/vocab.json
fredithefish-starfishrp-v4-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/fredithefish-starfishrp-v4_reward/merges.txt
fredithefish-starfishrp-v4-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/fredithefish-starfishrp-v4_reward/tokenizer.json
Job fredithefish-starfishrp-v4-mkmlizer completed after 97.92s with status: succeeded
Stopping job with name fredithefish-starfishrp-v4-mkmlizer
Pipeline stage MKMLizer completed in 102.43s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.46s
Running pipeline stage ISVCDeployer
Creating inference service fredithefish-starfishrp-v4
Waiting for inference service fredithefish-starfishrp-v4 to be ready
Inference service fredithefish-starfishrp-v4 ready after 40.22381830215454s
Pipeline stage ISVCDeployer completed in 48.04s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.6559038162231445s
Received healthy response to inference request in 1.0691323280334473s
Received healthy response to inference request in 1.1668121814727783s
Received healthy response to inference request in 1.0859143733978271s
Received healthy response to inference request in 1.0788893699645996s
5 requests
0 failed requests
5th percentile: 1.0710837364196777
10th percentile: 1.0730351448059081
20th percentile: 1.0769379615783692
30th percentile: 1.0802943706512451
40th percentile: 1.0831043720245361
50th percentile: 1.0859143733978271
60th percentile: 1.1182734966278076
70th percentile: 1.150632619857788
80th percentile: 1.2646305084228517
90th percentile: 1.4602671623229981
95th percentile: 1.5580854892730711
99th percentile: 1.63634015083313
mean time: 1.2113304138183594
Pipeline stage StressChecker completed in 7.16s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 0.04s
Running pipeline stage DaemonicSafetyScorer
Pipeline stage DaemonicSafetyScorer completed in 0.04s
Running M-Eval for topic stay_in_character
fredithefish-starfishrp_v4 status is now deployed due to DeploymentManager action
M-Eval Dataset for topic stay_in_character is loaded
fredithefish-starfishrp_v4 status is now inactive due to auto deactivation removed underperforming models

Usage Metrics

Latency Metrics