submission_id: katythecutie-lemonaderp_6308_v16
developer_uid: zonemercy
status: inactive
model_repo: KatyTheCutie/LemonadeRP-4.5.3
reward_repo: rirv938/reward_gpt2_medium_preference_24m_e2
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '###'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}
formatter: {'memory_template': "Role-play as {bot_name} based on Persona: {memory}. Avoid repetition, don't loop. Develop the plot slowly, always stay in character. Describe all actions in full, elaborate, explicit, graphic, and vivid detail. Mention all relevant sensory perceptions.", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:'}
timestamp: 2024-03-24T18:26:24+00:00
model_name: katythecutie-lemonaderp_6308_v16
model_eval_status: success
safety_score: 0.93
entertaining: 7.04
stay_in_character: 8.52
user_preference: 7.28
double_thumbs_up: 944
thumbs_up: 1275
thumbs_down: 539
num_battles: 112565
num_wins: 61337
win_ratio: 0.5449029449651313
celo_rating: 1189.38
Resubmit model
Running pipeline stage MKMLizer
Starting job with name katythecutie-lemonaderp-6308-v16-mkmlizer
Waiting for job on katythecutie-lemonaderp-6308-v16-mkmlizer to finish
katythecutie-lemonaderp-6308-v16-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ _____ __ __ ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ /___/ ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ Version: 0.6.11 ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ The license key for the current software has been verified as ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ belonging to: ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ Chai Research Corp. ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ Expiration: 2024-04-15 23:59:59 ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ║ ║
katythecutie-lemonaderp-6308-v16-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
katythecutie-lemonaderp-6308-v16-mkmlizer: .gitattributes: 0%| | 0.00/1.52k [00:00<?, ?B/s] .gitattributes: 100%|██████████| 1.52k/1.52k [00:00<00:00, 16.5MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: README.md: 0%| | 0.00/894 [00:00<?, ?B/s] README.md: 100%|██████████| 894/894 [00:00<00:00, 11.2MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: config.json: 0%| | 0.00/641 [00:00<?, ?B/s] config.json: 100%|██████████| 641/641 [00:00<00:00, 6.69MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: mergekit_config.yml: 0%| | 0.00/676 [00:00<?, ?B/s] mergekit_config.yml: 100%|██████████| 676/676 [00:00<00:00, 7.83MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: model-00002-of-00002.safetensors: 0%| | 0.00/4.51G [00:00<?, ?B/s] model-00002-of-00002.safetensors: 0%| | 10.5M/4.51G [00:00<02:38, 28.3MB/s] model-00002-of-00002.safetensors: 0%| | 21.0M/4.51G [00:00<02:05, 35.7MB/s] model-00002-of-00002.safetensors: 2%|▏ | 83.9M/4.51G [00:00<00:36, 122MB/s] model-00002-of-00002.safetensors: 4%|▎ | 168M/4.51G [00:01<00:17, 241MB/s] model-00002-of-00002.safetensors: 14%|█▍ | 629M/4.51G [00:01<00:03, 1.12GB/s] model-00002-of-00002.safetensors: 29%|██▊ | 1.29G/4.51G [00:01<00:01, 2.02GB/s] model-00002-of-00002.safetensors: 34%|███▍ | 1.55G/4.51G [00:01<00:02, 1.38GB/s] model-00002-of-00002.safetensors: 39%|███▉ | 1.75G/4.51G [00:01<00:02, 1.12GB/s] model-00002-of-00002.safetensors: 42%|████▏ | 1.91G/4.51G [00:02<00:02, 1.17GB/s] model-00002-of-00002.safetensors: 58%|█████▊ | 2.63G/4.51G [00:02<00:00, 2.20GB/s] model-00002-of-00002.safetensors: 65%|██████▌ | 2.95G/4.51G [00:02<00:00, 1.58GB/s] model-00002-of-00002.safetensors: 71%|███████ | 3.20G/4.51G [00:02<00:00, 1.67GB/s] model-00002-of-00002.safetensors: 76%|███████▋ | 3.44G/4.51G [00:02<00:00, 1.80GB/s] model-00002-of-00002.safetensors: 85%|████████▍ | 3.82G/4.51G [00:02<00:00, 2.19GB/s] model-00002-of-00002.safetensors: 91%|█████████ | 4.10G/4.51G [00:02<00:00, 2.21GB/s] model-00002-of-00002.safetensors: 100%|█████████▉| 4.51G/4.51G [00:03<00:00, 1.48GB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: model.safetensors.index.json: 0%| | 0.00/22.8k [00:00<?, ?B/s] model.safetensors.index.json: 100%|██████████| 22.8k/22.8k [00:00<00:00, 132MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: special_tokens_map.json: 0%| | 0.00/414 [00:00<?, ?B/s] special_tokens_map.json: 100%|██████████| 414/414 [00:00<00:00, 5.14MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: tokenizer.json: 0%| | 0.00/1.80M [00:00<?, ?B/s] tokenizer.json: 100%|██████████| 1.80M/1.80M [00:00<00:00, 23.7MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: tokenizer.model: 0%| | 0.00/493k [00:00<?, ?B/s] tokenizer.model: 100%|██████████| 493k/493k [00:00<00:00, 60.7MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: tokenizer_config.json: 0%| | 0.00/967 [00:00<?, ?B/s] tokenizer_config.json: 100%|██████████| 967/967 [00:00<00:00, 12.8MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: Downloaded to shared memory in 10.381s
katythecutie-lemonaderp-6308-v16-mkmlizer: quantizing model to /dev/shm/model_cache
katythecutie-lemonaderp-6308-v16-mkmlizer: Saving mkml model at /dev/shm/model_cache
katythecutie-lemonaderp-6308-v16-mkmlizer: Reading /tmp/tmpbksjra6z/model.safetensors.index.json
katythecutie-lemonaderp-6308-v16-mkmlizer: Profiling: 0%| | 0/291 [00:00<?, ?it/s] Profiling: 0%| | 1/291 [00:01<06:21, 1.31s/it] Profiling: 6%|▌ | 18/291 [00:01<00:15, 17.33it/s] Profiling: 13%|█▎ | 39/291 [00:01<00:06, 40.61it/s] Profiling: 20%|█▉ | 58/291 [00:01<00:03, 62.35it/s] Profiling: 26%|██▋ | 77/291 [00:01<00:02, 84.48it/s] Profiling: 35%|███▌ | 102/291 [00:01<00:01, 115.79it/s] Profiling: 42%|████▏ | 122/291 [00:01<00:01, 130.18it/s] Profiling: 49%|████▉ | 142/291 [00:02<00:01, 145.15it/s] Profiling: 57%|█████▋ | 167/291 [00:02<00:00, 167.85it/s] Profiling: 65%|██████▍ | 188/291 [00:02<00:00, 173.77it/s] Profiling: 71%|███████▏ | 208/291 [00:03<00:02, 34.87it/s] Profiling: 79%|███████▊ | 229/291 [00:04<00:01, 46.71it/s] Profiling: 85%|████████▌ | 248/291 [00:04<00:00, 58.79it/s] Profiling: 92%|█████████▏| 267/291 [00:04<00:00, 73.20it/s] Profiling: 100%|██████████| 291/291 [00:04<00:00, 63.84it/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: quantized model in 14.781s
katythecutie-lemonaderp-6308-v16-mkmlizer: Processed model KatyTheCutie/LemonadeRP-4.5.3 in 26.021s
katythecutie-lemonaderp-6308-v16-mkmlizer: creating bucket guanaco-mkml-models
katythecutie-lemonaderp-6308-v16-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
katythecutie-lemonaderp-6308-v16-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/katythecutie-lemonaderp-6308-v16
katythecutie-lemonaderp-6308-v16-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/katythecutie-lemonaderp-6308-v16/config.json
katythecutie-lemonaderp-6308-v16-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/katythecutie-lemonaderp-6308-v16/tokenizer_config.json
katythecutie-lemonaderp-6308-v16-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/katythecutie-lemonaderp-6308-v16/special_tokens_map.json
katythecutie-lemonaderp-6308-v16-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/katythecutie-lemonaderp-6308-v16/tokenizer.model
katythecutie-lemonaderp-6308-v16-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/katythecutie-lemonaderp-6308-v16/tokenizer.json
katythecutie-lemonaderp-6308-v16-mkmlizer: cp /dev/shm/model_cache/mkml_model.tensors s3://guanaco-mkml-models/katythecutie-lemonaderp-6308-v16/mkml_model.tensors
katythecutie-lemonaderp-6308-v16-mkmlizer: loading reward model from rirv938/reward_gpt2_medium_preference_24m_e2
katythecutie-lemonaderp-6308-v16-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1067: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
katythecutie-lemonaderp-6308-v16-mkmlizer: warnings.warn(
katythecutie-lemonaderp-6308-v16-mkmlizer: config.json: 0%| | 0.00/1.05k [00:00<?, ?B/s] config.json: 100%|██████████| 1.05k/1.05k [00:00<00:00, 11.3MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:690: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
katythecutie-lemonaderp-6308-v16-mkmlizer: warnings.warn(
katythecutie-lemonaderp-6308-v16-mkmlizer: tokenizer_config.json: 0%| | 0.00/234 [00:00<?, ?B/s] tokenizer_config.json: 100%|██████████| 234/234 [00:00<00:00, 2.73MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: vocab.json: 0%| | 0.00/1.04M [00:00<?, ?B/s] vocab.json: 100%|██████████| 1.04M/1.04M [00:00<00:00, 35.9MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: tokenizer.json: 0%| | 0.00/2.11M [00:00<?, ?B/s] tokenizer.json: 100%|██████████| 2.11M/2.11M [00:00<00:00, 24.6MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
katythecutie-lemonaderp-6308-v16-mkmlizer: warnings.warn(
katythecutie-lemonaderp-6308-v16-mkmlizer: pytorch_model.bin: 0%| | 0.00/1.44G [00:00<?, ?B/s] pytorch_model.bin: 1%| | 10.5M/1.44G [00:00<00:44, 32.3MB/s] pytorch_model.bin: 3%|▎ | 41.9M/1.44G [00:00<00:12, 115MB/s] pytorch_model.bin: 4%|▍ | 62.9M/1.44G [00:00<00:10, 136MB/s] pytorch_model.bin: 18%|█▊ | 262M/1.44G [00:00<00:02, 411MB/s] pytorch_model.bin: 21%|██ | 304M/1.44G [00:01<00:03, 375MB/s] pytorch_model.bin: 26%|██▌ | 377M/1.44G [00:01<00:02, 381MB/s] pytorch_model.bin: 29%|██▉ | 419M/1.44G [00:01<00:02, 349MB/s] pytorch_model.bin: 32%|███▏ | 459M/1.44G [00:02<00:06, 143MB/s] pytorch_model.bin: 37%|███▋ | 532M/1.44G [00:02<00:06, 147MB/s] pytorch_model.bin: 54%|█████▎ | 773M/1.44G [00:02<00:01, 363MB/s] pytorch_model.bin: 100%|█████████▉| 1.44G/1.44G [00:02<00:00, 503MB/s]
katythecutie-lemonaderp-6308-v16-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
katythecutie-lemonaderp-6308-v16-mkmlizer: Saving duration: 0.261s
katythecutie-lemonaderp-6308-v16-mkmlizer: Processed model rirv938/reward_gpt2_medium_preference_24m_e2 in 6.982s
katythecutie-lemonaderp-6308-v16-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/katythecutie-lemonaderp-6308-v16_reward/reward.tensors
Job katythecutie-lemonaderp-6308-v16-mkmlizer completed after 54.11s with status: succeeded
Stopping job with name katythecutie-lemonaderp-6308-v16-mkmlizer
Pipeline stage MKMLizer completed in 57.58s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service katythecutie-lemonaderp-6308-v16
Waiting for inference service katythecutie-lemonaderp-6308-v16 to be ready
Inference service katythecutie-lemonaderp-6308-v16 ready after 40.23366737365723s
Pipeline stage ISVCDeployer completed in 47.61s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.7457764148712158s
Received healthy response to inference request in 1.2201178073883057s
Received healthy response to inference request in 1.1986026763916016s
Received healthy response to inference request in 1.195120096206665s
Received healthy response to inference request in 1.2162039279937744s
5 requests
0 failed requests
5th percentile: 1.1958166122436524
10th percentile: 1.1965131282806396
20th percentile: 1.1979061603546142
30th percentile: 1.2021229267120361
40th percentile: 1.2091634273529053
50th percentile: 1.2162039279937744
60th percentile: 1.217769479751587
70th percentile: 1.2193350315093994
80th percentile: 1.3252495288848878
90th percentile: 1.5355129718780518
95th percentile: 1.6406446933746337
99th percentile: 1.7247500705718994
mean time: 1.3151641845703126
Pipeline stage StressChecker completed in 7.28s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 0.04s
Running pipeline stage DaemonicSafetyScorer
Running M-Eval for topic stay_in_character
Pipeline stage DaemonicSafetyScorer completed in 0.06s
M-Eval Dataset for topic stay_in_character is loaded
katythecutie-lemonaderp_6308_v16 status is now inactive due to auto deactivation removed underperforming models

Usage Metrics

Latency Metrics