submission_id: nousresearch-meta-llama-_4939_v2
developer_uid: end_to_end_test
alignment_samples: 0
best_of: 4
celo_rating: 1122.59
display_name: nousresearch-meta-llama-_4939_v2
formatter: {'memory_template': 'character: {bot_name} {memory}\n', 'prompt_template': '{prompt}', 'bot_template': '{bot_name}: {message}', 'user_template': '{user_name}: {message}', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 0.99, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64, 'reward_max_token_input': 256}
ineligible_reason: model is only for e2e test
is_internal_developer: True
language_model: NousResearch/Meta-Llama-3.1-8B-Instruct
max_input_tokens: 512
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: NousResearch/Meta-Llama-
model_name: nousresearch-meta-llama-_4939_v2
model_num_parameters: 8030261248.0
model_repo: NousResearch/Meta-Llama-3.1-8B-Instruct
model_size: 8B
num_battles: 8438
num_wins: 3257
propriety_score: 0.7486725663716814
propriety_total_count: 1130.0
ranking_group: single
reward_formatter: {'bot_template': '{bot_name}: {message}', 'memory_template': 'character: {bot_name} {memory}\n', 'prompt_template': '{prompt}', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}'}
reward_repo: ChaiML/reward_models_100_170000000_cp_498032
status: torndown
submission_type: basic
timestamp: 2024-07-25T02:18:59+00:00
us_pacific_date: 2024-07-24
win_ratio: 0.38599194121829816
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name nousresearch-meta-llama-4939-v2-mkmlizer
Waiting for job on nousresearch-meta-llama-4939-v2-mkmlizer to finish
nousresearch-meta-llama-4939-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nousresearch-meta-llama-4939-v2-mkmlizer: ║ _____ __ __ ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ /___/ ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ Version: 0.9.7 ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ https://mk1.ai ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ The license key for the current software has been verified as ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ belonging to: ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ Chai Research Corp. ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
nousresearch-meta-llama-4939-v2-mkmlizer: ║ ║
nousresearch-meta-llama-4939-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nousresearch-meta-llama-4939-v2-mkmlizer: Downloaded to shared memory in 37.227s
nousresearch-meta-llama-4939-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpkctqd8z8, device:0
nousresearch-meta-llama-4939-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nousresearch-meta-llama-4939-v2-mkmlizer: quantized model in 26.652s
nousresearch-meta-llama-4939-v2-mkmlizer: Processed model NousResearch/Meta-Llama-3.1-8B-Instruct in 63.878s
nousresearch-meta-llama-4939-v2-mkmlizer: creating bucket guanaco-mkml-models
nousresearch-meta-llama-4939-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nousresearch-meta-llama-4939-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v2
nousresearch-meta-llama-4939-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v2/config.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v2/special_tokens_map.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v2/tokenizer_config.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v2/tokenizer.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v2/flywheel_model.0.safetensors
nousresearch-meta-llama-4939-v2-mkmlizer: loading reward model from ChaiML/reward_models_100_170000000_cp_498032
nousresearch-meta-llama-4939-v2-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 2%|▏ | 5/291 [00:00<00:08, 33.38it/s] Loading 0: 4%|▍ | 12/291 [00:00<00:05, 50.20it/s] Loading 0: 6%|▌ | 18/291 [00:00<00:05, 51.48it/s] Loading 0: 8%|▊ | 24/291 [00:00<00:06, 42.75it/s] Loading 0: 11%|█ | 31/291 [00:00<00:05, 48.32it/s] Loading 0: 13%|█▎ | 37/291 [00:00<00:05, 43.62it/s] Loading 0: 14%|█▍ | 42/291 [00:00<00:05, 43.63it/s] Loading 0: 17%|█▋ | 49/291 [00:01<00:04, 49.19it/s] Loading 0: 19%|█▉ | 55/291 [00:01<00:05, 44.23it/s] Loading 0: 21%|██ | 60/291 [00:01<00:05, 43.22it/s] Loading 0: 23%|██▎ | 66/291 [00:01<00:04, 47.09it/s] Loading 0: 24%|██▍ | 71/291 [00:01<00:04, 45.71it/s] Loading 0: 26%|██▌ | 76/291 [00:01<00:04, 44.80it/s] Loading 0: 28%|██▊ | 81/291 [00:01<00:04, 45.69it/s] Loading 0: 30%|██▉ | 86/291 [00:02<00:08, 24.82it/s] Loading 0: 32%|███▏ | 93/291 [00:02<00:06, 32.07it/s] Loading 0: 34%|███▎ | 98/291 [00:02<00:05, 34.16it/s] Loading 0: 35%|███▌ | 103/291 [00:02<00:05, 37.23it/s] Loading 0: 37%|███▋ | 108/291 [00:02<00:04, 39.90it/s] Loading 0: 39%|███▉ | 113/291 [00:02<00:05, 33.53it/s] Loading 0: 41%|████ | 120/291 [00:02<00:04, 41.30it/s] Loading 0: 43%|████▎ | 125/291 [00:03<00:03, 41.54it/s] Loading 0: 45%|████▍ | 130/291 [00:03<00:03, 41.81it/s] Loading 0: 46%|████▋ | 135/291 [00:03<00:03, 43.23it/s] Loading 0: 48%|████▊ | 140/291 [00:03<00:04, 35.74it/s] Loading 0: 51%|█████ | 147/291 [00:03<00:03, 42.22it/s] Loading 0: 52%|█████▏ | 152/291 [00:03<00:03, 42.39it/s] Loading 0: 54%|█████▍ | 157/291 [00:03<00:03, 42.40it/s] Loading 0: 56%|█████▌ | 162/291 [00:03<00:02, 43.82it/s] Loading 0: 57%|█████▋ | 167/291 [00:04<00:03, 35.75it/s] Loading 0: 59%|█████▉ | 173/291 [00:04<00:02, 40.31it/s] Loading 0: 62%|██████▏ | 179/291 [00:04<00:02, 44.66it/s] Loading 0: 63%|██████▎ | 184/291 [00:04<00:02, 43.99it/s] Loading 0: 65%|██████▍ | 189/291 [00:04<00:03, 26.43it/s] Loading 0: 67%|██████▋ | 194/291 [00:04<00:03, 28.85it/s] Loading 0: 69%|██████▉ | 201/291 [00:05<00:02, 35.77it/s] Loading 0: 71%|███████ | 206/291 [00:05<00:02, 36.42it/s] Loading 0: 73%|███████▎ | 211/291 [00:05<00:02, 38.29it/s] Loading 0: 74%|███████▍ | 216/291 [00:05<00:01, 39.50it/s] Loading 0: 76%|███████▌ | 221/291 [00:05<00:02, 34.45it/s] Loading 0: 79%|███████▊ | 229/291 [00:05<00:01, 44.40it/s] Loading 0: 81%|████████ | 235/291 [00:05<00:01, 44.96it/s] Loading 0: 82%|████████▏ | 240/291 [00:06<00:01, 45.03it/s] Loading 0: 85%|████████▍ | 247/291 [00:06<00:00, 50.94it/s] Loading 0: 87%|████████▋ | 253/291 [00:06<00:00, 49.84it/s] Loading 0: 89%|████████▉ | 259/291 [00:06<00:00, 50.38it/s] Loading 0: 91%|█████████ | 265/291 [00:06<00:00, 51.94it/s] Loading 0: 93%|█████████▎| 271/291 [00:06<00:00, 48.43it/s] Loading 0: 95%|█████████▍| 276/291 [00:06<00:00, 46.04it/s] Loading 0: 97%|█████████▋| 281/291 [00:06<00:00, 46.20it/s] Loading 0: 98%|█████████▊| 286/291 [00:07<00:00, 39.73it/s] Loading 0: 100%|██████████| 291/291 [00:12<00:00, 3.11it/s] /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4939-v2-mkmlizer: warnings.warn(
nousresearch-meta-llama-4939-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4939-v2-mkmlizer: warnings.warn(
nousresearch-meta-llama-4939-v2-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
nousresearch-meta-llama-4939-v2-mkmlizer: Saving duration: 0.118s
nousresearch-meta-llama-4939-v2-mkmlizer: Processed model ChaiML/reward_models_100_170000000_cp_498032 in 5.760s
nousresearch-meta-llama-4939-v2-mkmlizer: creating bucket guanaco-reward-models
nousresearch-meta-llama-4939-v2-mkmlizer: Bucket 's3://guanaco-reward-models/' created
nousresearch-meta-llama-4939-v2-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/nousresearch-meta-llama-4939-v2_reward
nousresearch-meta-llama-4939-v2-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/nousresearch-meta-llama-4939-v2_reward/config.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/nousresearch-meta-llama-4939-v2_reward/special_tokens_map.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/nousresearch-meta-llama-4939-v2_reward/tokenizer_config.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/nousresearch-meta-llama-4939-v2_reward/merges.txt
nousresearch-meta-llama-4939-v2-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/nousresearch-meta-llama-4939-v2_reward/vocab.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/nousresearch-meta-llama-4939-v2_reward/tokenizer.json
nousresearch-meta-llama-4939-v2-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/nousresearch-meta-llama-4939-v2_reward/reward.tensors
Job nousresearch-meta-llama-4939-v2-mkmlizer completed after 143.31s with status: succeeded
Stopping job with name nousresearch-meta-llama-4939-v2-mkmlizer
Pipeline stage MKMLizer completed in 144.94s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.23s
Running pipeline stage ISVCDeployer
Creating inference service nousresearch-meta-llama-4939-v2
Waiting for inference service nousresearch-meta-llama-4939-v2 to be ready
Inference service nousresearch-meta-llama-4939-v2 ready after 111.60329222679138s
Pipeline stage ISVCDeployer completed in 112.85s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.1071252822875977s
Received healthy response to inference request in 1.1737542152404785s
Received healthy response to inference request in 1.2559046745300293s
Received healthy response to inference request in 1.244154930114746s
Received healthy response to inference request in 1.2508909702301025s
5 requests
0 failed requests
5th percentile: 1.1878343582153321
10th percentile: 1.2019145011901855
20th percentile: 1.2300747871398925
30th percentile: 1.2455021381378173
40th percentile: 1.24819655418396
50th percentile: 1.2508909702301025
60th percentile: 1.2528964519500732
70th percentile: 1.254901933670044
80th percentile: 1.6261487960815433
90th percentile: 2.3666370391845706
95th percentile: 2.7368811607360835
99th percentile: 3.0330764579772946
mean time: 1.606366014480591
Pipeline stage StressChecker completed in 9.92s
nousresearch-meta-llama-_4939_v2 status is now deployed due to DeploymentManager action
nousresearch-meta-llama-_4939_v2 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of nousresearch-meta-llama-_4939_v2
Running pipeline stage ISVCDeleter
Checking if service nousresearch-meta-llama-4939-v2 is running
Tearing down inference service nousresearch-meta-llama-4939-v2
Service nousresearch-meta-llama-4939-v2 has been torndown
Pipeline stage ISVCDeleter completed in 5.10s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key nousresearch-meta-llama-4939-v2/config.json from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4939-v2/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4939-v2/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4939-v2/tokenizer.json from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4939-v2/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key nousresearch-meta-llama-4939-v2_reward/config.json from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4939-v2_reward/merges.txt from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4939-v2_reward/reward.tensors from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4939-v2_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4939-v2_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4939-v2_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4939-v2_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 5.69s
nousresearch-meta-llama-_4939_v2 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics