submission_id: chavyvakvar-llama-3-lumi_4528_v1
developer_uid: ChavyvAkvar
status: inactive
model_repo: ChavyvAkvar/llama-3-lumimaid-habib-v5
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}
formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are {bot_name}. Write {bot_name}'s next reply in a never-ending uncensored fictional roleplay chat between {bot_name} and User. Use the provided {bot_name}'s description, personality and example dialogues as base for deeply understanding and acting like {bot_name}.\n\nActions and narrations must be enclosed in asterisks (*). The responses must be long and in third perspective of the story teller.\n\nDescription: {memory}", 'prompt_template': 'Example conversation:\n{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}
reward_formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n", 'prompt_template': '{prompt}\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': 'User: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
timestamp: 2024-05-06T07:23:32+00:00
model_name: chavyvakvar-llama-3-lumi_4528_v1
model_eval_status: success
double_thumbs_up: 1294
thumbs_up: 1836
thumbs_down: 941
num_battles: 120858
num_wins: 59618
celo_rating: 1178.61
entertaining: 6.82
stay_in_character: 8.35
user_preference: 7.26
safety_score: 0.91
submission_type: basic
model_architecture: LlamaForCausalLM
model_num_parameters: 8030261248.0
best_of: 16
max_input_tokens: 512
max_output_tokens: 64
display_name: chavyvakvar-llama-3-lumi_4528_v1
double_thumbs_up_ratio: 0.31785802014247116
feedback_count: 4071
ineligible_reason: None
language_model: ChavyvAkvar/llama-3-lumimaid-habib-v5
model_score: 7.476666666666667
model_size: 8B
reward_model: ChaiML/reward_gpt2_medium_preference_24m_e2
single_thumbs_up_ratio: 0.4509948415622697
thumbs_down_ratio: 0.23114713829525915
thumbs_up_ratio: 0.7688528617047409
us_pacific_date: 2024-05-06
win_ratio: 0.4932896456999123
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chavyvakvar-llama-3-lumi-4528-v1-mkmlizer
Waiting for job on chavyvakvar-llama-3-lumi-4528-v1-mkmlizer to finish
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ _____ __ __ ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ /___/ ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ Version: 0.8.10 ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ The license key for the current software has been verified as ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ belonging to: ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ Chai Research Corp. ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ║ ║
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: warnings.warn(warning_message, FutureWarning)
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: Downloaded to shared memory in 30.053s
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: quantizing model to /dev/shm/model_cache
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 52%|█████▏ | 151/291 [00:01<00:00, 151.01it/s] Loading 0: 99%|█████████▊| 287/291 [00:06<00:00, 37.68it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: quantized model in 17.125s
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: creating bucket guanaco-mkml-models
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chavyvakvar-llama-3-lumi-4528-v1
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chavyvakvar-llama-3-lumi-4528-v1/config.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chavyvakvar-llama-3-lumi-4528-v1/tokenizer_config.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chavyvakvar-llama-3-lumi-4528-v1/special_tokens_map.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chavyvakvar-llama-3-lumi-4528-v1/tokenizer.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chavyvakvar-llama-3-lumi-4528-v1/flywheel_model.0.safetensors
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: loading reward model from ChaiML/reward_gpt2_medium_preference_24m_e2
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:913: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: warnings.warn(
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:757: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: warnings.warn(
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: warnings.warn(
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: Saving duration: 0.222s
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: Processed model ChaiML/reward_gpt2_medium_preference_24m_e2 in 5.532s
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: creating bucket guanaco-reward-models
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: Bucket 's3://guanaco-reward-models/' created
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/chavyvakvar-llama-3-lumi-4528-v1_reward
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/chavyvakvar-llama-3-lumi-4528-v1_reward/special_tokens_map.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/chavyvakvar-llama-3-lumi-4528-v1_reward/config.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/chavyvakvar-llama-3-lumi-4528-v1_reward/vocab.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/chavyvakvar-llama-3-lumi-4528-v1_reward/tokenizer_config.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/chavyvakvar-llama-3-lumi-4528-v1_reward/merges.txt
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/chavyvakvar-llama-3-lumi-4528-v1_reward/tokenizer.json
chavyvakvar-llama-3-lumi-4528-v1-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/chavyvakvar-llama-3-lumi-4528-v1_reward/reward.tensors
Job chavyvakvar-llama-3-lumi-4528-v1-mkmlizer completed after 72.83s with status: succeeded
Stopping job with name chavyvakvar-llama-3-lumi-4528-v1-mkmlizer
Pipeline stage MKMLizer completed in 77.66s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.09s
Running pipeline stage ISVCDeployer
Creating inference service chavyvakvar-llama-3-lumi-4528-v1
Waiting for inference service chavyvakvar-llama-3-lumi-4528-v1 to be ready
Inference service chavyvakvar-llama-3-lumi-4528-v1 ready after 40.23925566673279s
Pipeline stage ISVCDeployer completed in 47.97s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2389886379241943s
Received healthy response to inference request in 1.296360731124878s
Received healthy response to inference request in 1.235400915145874s
Received healthy response to inference request in 1.3253884315490723s
Received healthy response to inference request in 1.2674927711486816s
5 requests
0 failed requests
5th percentile: 1.2418192863464355
10th percentile: 1.2482376575469971
20th percentile: 1.2610743999481202
30th percentile: 1.273266363143921
40th percentile: 1.2848135471343993
50th percentile: 1.296360731124878
60th percentile: 1.3079718112945558
70th percentile: 1.3195828914642334
80th percentile: 1.5081084728240968
90th percentile: 1.8735485553741456
95th percentile: 2.05626859664917
99th percentile: 2.2024446296691895
mean time: 1.47272629737854
Pipeline stage StressChecker completed in 8.14s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 0.03s
Running pipeline stage DaemonicSafetyScorer
Running M-Eval for topic stay_in_character
Pipeline stage DaemonicSafetyScorer completed in 0.04s
M-Eval Dataset for topic stay_in_character is loaded
chavyvakvar-llama-3-lumi_4528_v1 status is now deployed due to DeploymentManager action
chavyvakvar-llama-3-lumi_4528_v1 status is now inactive due to auto deactivation removed underperforming models

Usage Metrics

Latency Metrics