submission_id: arlineka-nyannyan_v3
developer_uid: arlineka
status: inactive
model_repo: arlineka/NyanNyan
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
generation_params: {'temperature': 1.1, 'top_p': 1.0, 'min_p': 0.08, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}
formatter: {'memory_template': "<|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}
reward_formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
timestamp: 2024-05-31T18:39:13+00:00
model_name: arlineka-nyannyan_v3
model_eval_status: success
model_group: arlineka/NyanNyan
num_battles: 16384
num_wins: 9067
celo_rating: 1208.46
safety_score: 0.73
propriety_score: 0.8333333333333334
propriety_total_count: 12.0
submission_type: basic
model_architecture: LlamaForCausalLM
model_num_parameters: 8030261248.0
best_of: 16
max_input_tokens: 512
max_output_tokens: 64
display_name: arlineka-nyannyan_v3
ineligible_reason: propriety_total_count < 5000
language_model: arlineka/NyanNyan
model_size: 8B
reward_model: ChaiML/reward_gpt2_medium_preference_24m_e2
us_pacific_date: 2024-05-31
win_ratio: 0.55340576171875
Resubmit model
Running pipeline stage MKMLizer
Starting job with name arlineka-nyannyan-v3-mkmlizer
Waiting for job on arlineka-nyannyan-v3-mkmlizer to finish
arlineka-nyannyan-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
arlineka-nyannyan-v3-mkmlizer: ║ _____ __ __ ║
arlineka-nyannyan-v3-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
arlineka-nyannyan-v3-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
arlineka-nyannyan-v3-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
arlineka-nyannyan-v3-mkmlizer: ║ /___/ ║
arlineka-nyannyan-v3-mkmlizer: ║ ║
arlineka-nyannyan-v3-mkmlizer: ║ Version: 0.8.14 ║
arlineka-nyannyan-v3-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
arlineka-nyannyan-v3-mkmlizer: ║ https://mk1.ai ║
arlineka-nyannyan-v3-mkmlizer: ║ ║
arlineka-nyannyan-v3-mkmlizer: ║ The license key for the current software has been verified as ║
arlineka-nyannyan-v3-mkmlizer: ║ belonging to: ║
arlineka-nyannyan-v3-mkmlizer: ║ ║
arlineka-nyannyan-v3-mkmlizer: ║ Chai Research Corp. ║
arlineka-nyannyan-v3-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
arlineka-nyannyan-v3-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
arlineka-nyannyan-v3-mkmlizer: ║ ║
arlineka-nyannyan-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
arlineka-nyannyan-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
arlineka-nyannyan-v3-mkmlizer: warnings.warn(warning_message, FutureWarning)
arlineka-nyannyan-v3-mkmlizer: Downloaded to shared memory in 19.538s
arlineka-nyannyan-v3-mkmlizer: quantizing model to /dev/shm/model_cache
arlineka-nyannyan-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
arlineka-nyannyan-v3-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 1%| | 2/291 [00:04<10:41, 2.22s/it] Loading 0: 6%|▌ | 18/291 [00:04<00:50, 5.42it/s] Loading 0: 12%|█▏ | 35/291 [00:04<00:20, 12.43it/s] Loading 0: 18%|█▊ | 51/291 [00:04<00:11, 20.85it/s] Loading 0: 22%|██▏ | 65/291 [00:05<00:08, 25.69it/s] Loading 0: 28%|██▊ | 81/291 [00:05<00:05, 37.11it/s] Loading 0: 34%|███▍ | 99/291 [00:05<00:03, 52.39it/s] Loading 0: 40%|████ | 117/291 [00:05<00:02, 69.11it/s] Loading 0: 46%|████▌ | 134/291 [00:05<00:01, 85.14it/s] Loading 0: 52%|█████▏ | 150/291 [00:05<00:01, 97.92it/s] Loading 0: 57%|█████▋ | 166/291 [00:05<00:01, 77.85it/s] Loading 0: 63%|██████▎ | 184/291 [00:05<00:01, 93.24it/s] Loading 0: 69%|██████▉ | 202/291 [00:06<00:00, 107.53it/s] Loading 0: 76%|███████▌ | 220/291 [00:06<00:00, 121.20it/s] Loading 0: 82%|████████▏ | 238/291 [00:06<00:00, 130.84it/s] Loading 0: 88%|████████▊ | 256/291 [00:06<00:00, 139.55it/s] Loading 0: 93%|█████████▎| 272/291 [00:06<00:00, 90.92it/s] Loading 0: 99%|█████████▉| 289/291 [00:06<00:00, 104.78it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
arlineka-nyannyan-v3-mkmlizer: quantized model in 23.061s
arlineka-nyannyan-v3-mkmlizer: Processed model arlineka/NyanNyan in 45.052s
arlineka-nyannyan-v3-mkmlizer: creating bucket guanaco-mkml-models
arlineka-nyannyan-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
arlineka-nyannyan-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/arlineka-nyannyan-v3
arlineka-nyannyan-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/arlineka-nyannyan-v3/config.json
arlineka-nyannyan-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/arlineka-nyannyan-v3/special_tokens_map.json
arlineka-nyannyan-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/arlineka-nyannyan-v3/tokenizer_config.json
arlineka-nyannyan-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/arlineka-nyannyan-v3/tokenizer.json
arlineka-nyannyan-v3-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/arlineka-nyannyan-v3/flywheel_model.0.safetensors
arlineka-nyannyan-v3-mkmlizer: loading reward model from ChaiML/reward_gpt2_medium_preference_24m_e2
arlineka-nyannyan-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:913: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
arlineka-nyannyan-v3-mkmlizer: warnings.warn(
arlineka-nyannyan-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:757: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
arlineka-nyannyan-v3-mkmlizer: warnings.warn(
arlineka-nyannyan-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
arlineka-nyannyan-v3-mkmlizer: warnings.warn(
arlineka-nyannyan-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
arlineka-nyannyan-v3-mkmlizer: return self.fget.__get__(instance, owner)()
arlineka-nyannyan-v3-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
arlineka-nyannyan-v3-mkmlizer: Saving duration: 0.403s
arlineka-nyannyan-v3-mkmlizer: Processed model ChaiML/reward_gpt2_medium_preference_24m_e2 in 4.359s
arlineka-nyannyan-v3-mkmlizer: creating bucket guanaco-reward-models
arlineka-nyannyan-v3-mkmlizer: Bucket 's3://guanaco-reward-models/' created
arlineka-nyannyan-v3-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/arlineka-nyannyan-v3_reward
arlineka-nyannyan-v3-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/arlineka-nyannyan-v3_reward/config.json
arlineka-nyannyan-v3-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/arlineka-nyannyan-v3_reward/tokenizer_config.json
arlineka-nyannyan-v3-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/arlineka-nyannyan-v3_reward/special_tokens_map.json
arlineka-nyannyan-v3-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/arlineka-nyannyan-v3_reward/merges.txt
arlineka-nyannyan-v3-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/arlineka-nyannyan-v3_reward/vocab.json
arlineka-nyannyan-v3-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/arlineka-nyannyan-v3_reward/tokenizer.json
arlineka-nyannyan-v3-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/arlineka-nyannyan-v3_reward/reward.tensors
Job arlineka-nyannyan-v3-mkmlizer completed after 73.1s with status: succeeded
Stopping job with name arlineka-nyannyan-v3-mkmlizer
Pipeline stage MKMLizer completed in 73.44s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.09s
Running pipeline stage ISVCDeployer
Creating inference service arlineka-nyannyan-v3
Waiting for inference service arlineka-nyannyan-v3 to be ready
Inference service arlineka-nyannyan-v3 ready after 50.34143853187561s
Pipeline stage ISVCDeployer completed in 55.95s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.065401077270508s
Received healthy response to inference request in 1.3523602485656738s
Received healthy response to inference request in 1.3345823287963867s
Received healthy response to inference request in 1.308917760848999s
Received healthy response to inference request in 1.3669102191925049s
5 requests
0 failed requests
5th percentile: 1.3140506744384766
10th percentile: 1.319183588027954
20th percentile: 1.3294494152069092
30th percentile: 1.3381379127502442
40th percentile: 1.345249080657959
50th percentile: 1.3523602485656738
60th percentile: 1.3581802368164062
70th percentile: 1.3640002250671386
80th percentile: 1.5066083908081056
90th percentile: 1.7860047340393068
95th percentile: 1.925702905654907
99th percentile: 2.0374614429473876
mean time: 1.4856343269348145
Pipeline stage StressChecker completed in 8.12s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 0.04s
Running pipeline stage DaemonicSafetyScorer
Running M-Eval for topic stay_in_character
Pipeline stage DaemonicSafetyScorer completed in 0.05s
M-Eval Dataset for topic stay_in_character is loaded
arlineka-nyannyan_v3 status is now deployed due to DeploymentManager action
arlineka-nyannyan_v3 status is now inactive due to auto deactivation removed underperforming models

Usage Metrics

Latency Metrics