submission_id: maywell-pivot-0-1-evil-a_v1
developer_uid: windyheath
alignment_samples: 0
best_of: 4
celo_rating: 1149.86
display_name: maywell-pivot-0-1-evil-a_v1
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64}
ineligible_reason: propriety_total_count < 800
is_internal_developer: False
language_model: maywell/PiVoT-0.1-Evil-a
max_input_tokens: 512
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_eval_status: success
model_group: maywell/PiVoT-0.1-Evil-a
model_name: maywell-pivot-0-1-evil-a_v1
model_num_parameters: 7241748480.0
model_repo: maywell/PiVoT-0.1-Evil-a
model_size: 7B
num_battles: 9145
num_wins: 4287
propriety_score: 0.6398305084745762
propriety_total_count: 708.0
ranking_group: single
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
status: torndown
submission_type: basic
timestamp: 2024-06-07T20:27:32+00:00
us_pacific_date: 2024-06-07
win_ratio: 0.4687807545106616
Resubmit model
Running pipeline stage MKMLizer
Starting job with name maywell-pivot-0-1-evil-a-v1-mkmlizer
Waiting for job on maywell-pivot-0-1-evil-a-v1-mkmlizer to finish
maywell-pivot-0-1-evil-a-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ _____ __ __ ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ /___/ ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ Version: 0.8.14 ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ https://mk1.ai ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ The license key for the current software has been verified as ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ belonging to: ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ Chai Research Corp. ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ║ ║
maywell-pivot-0-1-evil-a-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
maywell-pivot-0-1-evil-a-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
maywell-pivot-0-1-evil-a-v1-mkmlizer: warnings.warn(warning_message, FutureWarning)
maywell-pivot-0-1-evil-a-v1-mkmlizer: Downloaded to shared memory in 30.704s
maywell-pivot-0-1-evil-a-v1-mkmlizer: quantizing model to /dev/shm/model_cache
maywell-pivot-0-1-evil-a-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
maywell-pivot-0-1-evil-a-v1-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 5%|▍ | 14/291 [00:00<00:02, 130.77it/s] Loading 0: 11%|█ | 32/291 [00:00<00:01, 156.95it/s] Loading 0: 16%|█▋ | 48/291 [00:00<00:01, 146.25it/s] Loading 0: 22%|██▏ | 63/291 [00:00<00:01, 136.42it/s] Loading 0: 26%|██▋ | 77/291 [00:00<00:01, 133.46it/s] Loading 0: 32%|███▏ | 92/291 [00:00<00:01, 136.13it/s] Loading 0: 36%|███▋ | 106/291 [00:01<00:02, 78.11it/s] Loading 0: 42%|████▏ | 122/291 [00:01<00:01, 92.65it/s] Loading 0: 48%|████▊ | 140/291 [00:01<00:01, 109.22it/s] Loading 0: 54%|█████▍ | 158/291 [00:01<00:01, 122.48it/s] Loading 0: 60%|██████ | 176/291 [00:01<00:00, 134.06it/s] Loading 0: 67%|██████▋ | 194/291 [00:01<00:00, 141.80it/s] Loading 0: 72%|███████▏ | 210/291 [00:03<00:02, 28.55it/s] Loading 0: 76%|███████▋ | 222/291 [00:03<00:01, 34.74it/s] Loading 0: 82%|████████▏ | 240/291 [00:03<00:01, 47.04it/s] Loading 0: 89%|████████▊ | 258/291 [00:03<00:00, 60.84it/s] Loading 0: 95%|█████████▍| 276/291 [00:03<00:00, 76.04it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
maywell-pivot-0-1-evil-a-v1-mkmlizer: quantized model in 14.494s
maywell-pivot-0-1-evil-a-v1-mkmlizer: Processed model maywell/PiVoT-0.1-Evil-a in 47.541s
maywell-pivot-0-1-evil-a-v1-mkmlizer: creating bucket guanaco-mkml-models
maywell-pivot-0-1-evil-a-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
maywell-pivot-0-1-evil-a-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/maywell-pivot-0-1-evil-a-v1
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/maywell-pivot-0-1-evil-a-v1/config.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/maywell-pivot-0-1-evil-a-v1/special_tokens_map.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/maywell-pivot-0-1-evil-a-v1/tokenizer_config.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/maywell-pivot-0-1-evil-a-v1/tokenizer.model
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-mkml-models/maywell-pivot-0-1-evil-a-v1/added_tokens.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/maywell-pivot-0-1-evil-a-v1/tokenizer.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/maywell-pivot-0-1-evil-a-v1/flywheel_model.0.safetensors
maywell-pivot-0-1-evil-a-v1-mkmlizer: loading reward model from ChaiML/reward_gpt2_medium_preference_24m_e2
maywell-pivot-0-1-evil-a-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:913: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
maywell-pivot-0-1-evil-a-v1-mkmlizer: warnings.warn(
maywell-pivot-0-1-evil-a-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:757: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
maywell-pivot-0-1-evil-a-v1-mkmlizer: warnings.warn(
maywell-pivot-0-1-evil-a-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
maywell-pivot-0-1-evil-a-v1-mkmlizer: warnings.warn(
maywell-pivot-0-1-evil-a-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
maywell-pivot-0-1-evil-a-v1-mkmlizer: return self.fget.__get__(instance, owner)()
maywell-pivot-0-1-evil-a-v1-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
maywell-pivot-0-1-evil-a-v1-mkmlizer: Saving duration: 0.391s
maywell-pivot-0-1-evil-a-v1-mkmlizer: Processed model ChaiML/reward_gpt2_medium_preference_24m_e2 in 7.662s
maywell-pivot-0-1-evil-a-v1-mkmlizer: creating bucket guanaco-reward-models
maywell-pivot-0-1-evil-a-v1-mkmlizer: Bucket 's3://guanaco-reward-models/' created
maywell-pivot-0-1-evil-a-v1-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/maywell-pivot-0-1-evil-a-v1_reward
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/maywell-pivot-0-1-evil-a-v1_reward/tokenizer_config.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/maywell-pivot-0-1-evil-a-v1_reward/special_tokens_map.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/maywell-pivot-0-1-evil-a-v1_reward/config.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/maywell-pivot-0-1-evil-a-v1_reward/merges.txt
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/maywell-pivot-0-1-evil-a-v1_reward/vocab.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/maywell-pivot-0-1-evil-a-v1_reward/tokenizer.json
maywell-pivot-0-1-evil-a-v1-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/maywell-pivot-0-1-evil-a-v1_reward/reward.tensors
Job maywell-pivot-0-1-evil-a-v1-mkmlizer completed after 73.1s with status: succeeded
Stopping job with name maywell-pivot-0-1-evil-a-v1-mkmlizer
Pipeline stage MKMLizer completed in 74.03s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.08s
Running pipeline stage ISVCDeployer
Creating inference service maywell-pivot-0-1-evil-a-v1
Waiting for inference service maywell-pivot-0-1-evil-a-v1 to be ready
Inference service maywell-pivot-0-1-evil-a-v1 ready after 30.262393951416016s
Pipeline stage ISVCDeployer completed in 36.09s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.8080155849456787s
Received healthy response to inference request in 0.8442816734313965s
Received healthy response to inference request in 1.0690295696258545s
Received healthy response to inference request in 1.0811395645141602s
Received healthy response to inference request in 1.075742244720459s
5 requests
0 failed requests
5th percentile: 0.8892312526702881
10th percentile: 0.9341808319091797
20th percentile: 1.0240799903869628
30th percentile: 1.0703721046447754
40th percentile: 1.0730571746826172
50th percentile: 1.075742244720459
60th percentile: 1.0779011726379395
70th percentile: 1.08006010055542
80th percentile: 1.226514768600464
90th percentile: 1.5172651767730714
95th percentile: 1.6626403808593748
99th percentile: 1.7789405441284178
mean time: 1.1756417274475097
Pipeline stage StressChecker completed in 6.56s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 0.03s
Running M-Eval for topic stay_in_character
Running pipeline stage DaemonicSafetyScorer
M-Eval Dataset for topic stay_in_character is loaded
Pipeline stage DaemonicSafetyScorer completed in 0.07s
maywell-pivot-0-1-evil-a_v1 status is now deployed due to DeploymentManager action
maywell-pivot-0-1-evil-a_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of maywell-pivot-0-1-evil-a_v1
Running pipeline stage ISVCDeleter
Checking if service maywell-pivot-0-1-evil-a-v1 is running
Skipping teardown as no inference service was found
Pipeline stage ISVCDeleter completed in 2.23s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key maywell-pivot-0-1-evil-a-v1/added_tokens.json from bucket guanaco-mkml-models
Deleting key maywell-pivot-0-1-evil-a-v1/config.json from bucket guanaco-mkml-models
Deleting key maywell-pivot-0-1-evil-a-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key maywell-pivot-0-1-evil-a-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key maywell-pivot-0-1-evil-a-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key maywell-pivot-0-1-evil-a-v1/tokenizer.model from bucket guanaco-mkml-models
Deleting key maywell-pivot-0-1-evil-a-v1/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key maywell-pivot-0-1-evil-a-v1_reward/config.json from bucket guanaco-reward-models
Deleting key maywell-pivot-0-1-evil-a-v1_reward/merges.txt from bucket guanaco-reward-models
Deleting key maywell-pivot-0-1-evil-a-v1_reward/reward.tensors from bucket guanaco-reward-models
Deleting key maywell-pivot-0-1-evil-a-v1_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key maywell-pivot-0-1-evil-a-v1_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key maywell-pivot-0-1-evil-a-v1_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key maywell-pivot-0-1-evil-a-v1_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 3.85s
maywell-pivot-0-1-evil-a_v1 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics