submission_id: neversleep-noromaid-v0-_8068_v21
developer_uid: chai_backend_admin
status: torndown
model_repo: NeverSleep/Noromaid-v0.1-mixtral-8x7b-Instruct-v3
reward_repo: rirv938/reward_gpt2_medium_preference_24m_e2
generation_params: {'temperature': 0.8, 'top_p': 0.9, 'min_p': 0.0, 'top_k': 70, 'presence_penalty': 0.9, 'frequency_penalty': 0.9, 'stopping_words': ['\n', '</s>', '<|user|>', '###'], 'max_input_tokens': 512, 'best_of': 8, 'max_output_tokens': 128}
formatter: {'memory_template': '<s>[INST] This is an entertaining conversation. You are {bot_name} who has the persona: {memory}.\nEngage in a chat with {user_name} while staying in character. Try to flirt with {user_name}. Engage in *roleplay* actions. Describe the scene dramatically. \n', 'prompt_template': '{prompt}\n', 'bot_template': '{bot_name}: {message}</s>', 'user_template': '[INST] {user_name}: {message} [/INST]', 'response_template': '{bot_name}:', 'truncate_by_message': False}
reward_formatter: {'memory_template': 'Memory: {memory}\n', 'prompt_template': '{prompt}\n', 'bot_template': 'Bot: {message}\n', 'user_template': 'User: {message}\n', 'response_template': 'Bot:', 'truncate_by_message': False}
timestamp: 2024-03-31T02:15:38+00:00
model_name: Noromaid-v0-1-128
model_eval_status: success
model_group: NeverSleep/Noromaid-v0.1
num_battles: 317860
num_wins: 189483
celo_rating: 1224.91
propriety_score: 0.0
propriety_total_count: 0.0
submission_type: basic
model_architecture: MixtralForCausalLM
model_num_parameters: 46702792704.0
best_of: 8
max_input_tokens: 512
max_output_tokens: 128
display_name: Noromaid-v0-1-128
ineligible_reason: max_output_tokens!=64
language_model: NeverSleep/Noromaid-v0.1-mixtral-8x7b-Instruct-v3
model_size: 47B
reward_model: rirv938/reward_gpt2_medium_preference_24m_e2
us_pacific_date: 2024-03-30
win_ratio: 0.5961209337444158
preference_data_url: None
Resubmit model
Running pipeline stage MKMLizer
Starting job with name neversleep-noromaid-v0-8068-v21-mkmlizer
Waiting for job on neversleep-noromaid-v0-8068-v21-mkmlizer to finish
neversleep-noromaid-v0-8068-v21-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ _____ __ __ ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ /___/ ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ Version: 0.8.6 ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ The license key for the current software has been verified as ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ belonging to: ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ Chai Research Corp. ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ║ ║
neversleep-noromaid-v0-8068-v21-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
neversleep-noromaid-v0-8068-v21-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
neversleep-noromaid-v0-8068-v21-mkmlizer: warnings.warn(warning_message, FutureWarning)
Failed to get response for submission anhnv125-llama-op-v17-1_v27: ('http://anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40450->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
neversleep-noromaid-v0-8068-v21-mkmlizer: Downloaded to shared memory in 160.938s
neversleep-noromaid-v0-8068-v21-mkmlizer: quantizing model to /dev/shm/model_cache
neversleep-noromaid-v0-8068-v21-mkmlizer: Saving flywheel model at /dev/shm/model_cache
neversleep-noromaid-v0-8068-v21-mkmlizer: quantized model in 60.960s
neversleep-noromaid-v0-8068-v21-mkmlizer: Processed model NeverSleep/Noromaid-v0.1-mixtral-8x7b-Instruct-v3 in 229.311s
neversleep-noromaid-v0-8068-v21-mkmlizer: creating bucket guanaco-mkml-models
neversleep-noromaid-v0-8068-v21-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
neversleep-noromaid-v0-8068-v21-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/config.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/tokenizer_config.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/tokenizer.model
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/special_tokens_map.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/tokenizer.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/flywheel_model.1.safetensors
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/flywheel_model.3.safetensors
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/flywheel_model.0.safetensors
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v21/flywheel_model.2.safetensors
neversleep-noromaid-v0-8068-v21-mkmlizer: loading reward model from rirv938/reward_gpt2_medium_preference_24m_e2
neversleep-noromaid-v0-8068-v21-mkmlizer: Loading 0: 0%| | 0/995 [00:00<?, ?it/s] Loading 0: 5%|▌ | 52/995 [00:01<00:21, 43.92it/s] Loading 0: 11%|█ | 107/995 [00:02<00:19, 45.53it/s] Loading 0: 16%|█▋ | 162/995 [00:03<00:17, 47.82it/s] Loading 0: 21%|██ | 210/995 [00:04<00:17, 45.53it/s] Loading 0: 27%|██▋ | 265/995 [00:05<00:15, 46.56it/s] Loading 0: 28%|██▊ | 278/995 [00:15<01:14, 9.56it/s] Loading 0: 32%|███▏ | 320/995 [00:17<00:54, 12.32it/s] Loading 0: 37%|███▋ | 368/995 [00:18<00:38, 16.17it/s] Loading 0: 43%|████▎ | 423/995 [00:19<00:26, 21.37it/s] Loading 0: 48%|████▊ | 478/995 [00:20<00:19, 26.63it/s] Loading 0: 53%|█████▎ | 526/995 [00:21<00:15, 30.18it/s] Loading 0: 57%|█████▋ | 564/995 [00:32<00:14, 30.18it/s] Loading 0: 57%|█████▋ | 565/995 [00:32<00:40, 10.64it/s] Loading 0: 58%|█████▊ | 581/995 [00:33<00:38, 10.75it/s] Loading 0: 64%|██████▍ | 636/995 [00:34<00:23, 15.51it/s] Loading 0: 69%|██████▉ | 691/995 [00:35<00:14, 20.54it/s] Loading 0: 74%|███████▍ | 739/995 [00:36<00:10, 24.84it/s] Loading 0: 80%|███████▉ | 794/995 [00:37<00:06, 30.59it/s] Loading 0: 85%|████████▍ | 845/995 [00:48<00:04, 30.59it/s] Loading 0: 85%|████████▌ | 846/995 [00:48<00:12, 11.85it/s] Loading 0: 85%|████████▌ | 849/995 [00:49<00:13, 10.83it/s] Loading 0: 90%|█████████ | 897/995 [00:50<00:06, 15.11it/s] Loading 0: 96%|█████████▌| 952/995 [00:51<00:02, 21.15it/s] Loading 0: 100%|██████████| 995/995 [00:52<00:00, 23.91it/s] /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1096: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
neversleep-noromaid-v0-8068-v21-mkmlizer: warnings.warn(
neversleep-noromaid-v0-8068-v21-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:720: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
neversleep-noromaid-v0-8068-v21-mkmlizer: warnings.warn(
neversleep-noromaid-v0-8068-v21-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:466: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
neversleep-noromaid-v0-8068-v21-mkmlizer: warnings.warn(
neversleep-noromaid-v0-8068-v21-mkmlizer: /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
neversleep-noromaid-v0-8068-v21-mkmlizer: return self.fget.__get__(instance, owner)()
neversleep-noromaid-v0-8068-v21-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
neversleep-noromaid-v0-8068-v21-mkmlizer: Saving duration: 0.288s
neversleep-noromaid-v0-8068-v21-mkmlizer: Processed model rirv938/reward_gpt2_medium_preference_24m_e2 in 3.964s
neversleep-noromaid-v0-8068-v21-mkmlizer: creating bucket guanaco-reward-models
neversleep-noromaid-v0-8068-v21-mkmlizer: Bucket 's3://guanaco-reward-models/' created
neversleep-noromaid-v0-8068-v21-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v21_reward
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v21_reward/special_tokens_map.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v21_reward/config.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v21_reward/vocab.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v21_reward/merges.txt
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v21_reward/tokenizer_config.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v21_reward/tokenizer.json
neversleep-noromaid-v0-8068-v21-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v21_reward/reward.tensors
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Job neversleep-noromaid-v0-8068-v21-mkmlizer completed after 363.41s with status: succeeded
Stopping job with name neversleep-noromaid-v0-8068-v21-mkmlizer
Pipeline stage MKMLizer completed in 367.92s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.13s
Running pipeline stage ISVCDeployer
Creating inference service neversleep-noromaid-v0-8068-v21
Waiting for inference service neversleep-noromaid-v0-8068-v21 to be ready
Inference service neversleep-noromaid-v0-8068-v21 ready after 50.27430987358093s
Pipeline stage ISVCDeployer completed in 59.32s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.339120864868164s
Received healthy response to inference request in 3.7814245223999023s
Received healthy response to inference request in 2.9079976081848145s
Received healthy response to inference request in 3.1809592247009277s
Received healthy response to inference request in 4.412293910980225s
5 requests
0 failed requests
5th percentile: 2.9625899314880373
10th percentile: 3.0171822547912597
20th percentile: 3.126366901397705
30th percentile: 3.301052284240723
40th percentile: 3.5412384033203126
50th percentile: 3.7814245223999023
60th percentile: 4.004503059387207
70th percentile: 4.227581596374511
80th percentile: 4.353755474090576
90th percentile: 4.3830246925354
95th percentile: 4.3976593017578125
99th percentile: 4.4093669891357425
mean time: 3.724359226226807
Pipeline stage StressChecker completed in 19.31s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 0.04s
Running pipeline stage DaemonicSafetyScorer
Running M-Eval for topic stay_in_character
Pipeline stage DaemonicSafetyScorer completed in 0.04s
M-Eval Dataset for topic stay_in_character is loaded
neversleep-noromaid-v0-_8068_v21 status is now deployed due to DeploymentManager action
neversleep-noromaid-v0-_8068_v21 status is now inactive due to auto deactivation removed underperforming models
neversleep-noromaid-v0-_8068_v21 status is now deployed due to admin request
neversleep-noromaid-v0-_8068_v21 status is now inactive due to auto deactivation removed underperforming models
neversleep-noromaid-v0-_8068_v21 status is now deployed due to admin request
neversleep-noromaid-v0-_8068_v21 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of neversleep-noromaid-v0-_8068_v21
Running pipeline stage ISVCDeleter
Checking if service neversleep-noromaid-v0-8068-v21 is running
Tearing down inference service neversleep-noromaid-v0-8068-v21
Toredown service neversleep-noromaid-v0-8068-v21
Pipeline stage ISVCDeleter completed in 4.49s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key neversleep-noromaid-v0-8068-v21/config.json from bucket guanaco-mkml-models
Deleting key neversleep-noromaid-v0-8068-v21/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key neversleep-noromaid-v0-8068-v21/flywheel_model.1.safetensors from bucket guanaco-mkml-models
Deleting key neversleep-noromaid-v0-8068-v21/flywheel_model.2.safetensors from bucket guanaco-mkml-models
Deleting key neversleep-noromaid-v0-8068-v21/flywheel_model.3.safetensors from bucket guanaco-mkml-models
Deleting key neversleep-noromaid-v0-8068-v21/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key neversleep-noromaid-v0-8068-v21/tokenizer.json from bucket guanaco-mkml-models
Deleting key neversleep-noromaid-v0-8068-v21/tokenizer.model from bucket guanaco-mkml-models
Deleting key neversleep-noromaid-v0-8068-v21/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key neversleep-noromaid-v0-8068-v21_reward/config.json from bucket guanaco-reward-models
Deleting key neversleep-noromaid-v0-8068-v21_reward/merges.txt from bucket guanaco-reward-models
Deleting key neversleep-noromaid-v0-8068-v21_reward/reward.tensors from bucket guanaco-reward-models
Deleting key neversleep-noromaid-v0-8068-v21_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key neversleep-noromaid-v0-8068-v21_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key neversleep-noromaid-v0-8068-v21_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key neversleep-noromaid-v0-8068-v21_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 5.72s
neversleep-noromaid-v0-_8068_v21 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics