submission_id: nousresearch-meta-llama_4941_v76
developer_uid: robert_irvine
status: inactive
model_repo: NousResearch/Meta-Llama-3-8B-Instruct
reward_repo: ChaiML/gpt2_medium_pairwise_60m_step_937500
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|user|>', '###', '\n'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
reward_formatter: {'memory_template': 'Memory: {memory}\n', 'prompt_template': '{prompt}\n', 'bot_template': 'Bot: {message}\n', 'user_template': 'User: {message}\n', 'response_template': 'Bot:', 'truncate_by_message': False}
timestamp: 2024-07-03T18:25:16+00:00
model_name: nousresearch-meta-llama_4941_v76
model_group: NousResearch/Meta-Llama-
num_battles: 14023
num_wins: 6794
celo_rating: 1159.34
propriety_score: 0.7472381794078656
propriety_total_count: 6789.0
submission_type: basic
model_architecture: LlamaForCausalLM
model_num_parameters: 8030261248.0
best_of: 4
max_input_tokens: 512
max_output_tokens: 64
display_name: nousresearch-meta-llama_4941_v76
ineligible_reason: None
language_model: NousResearch/Meta-Llama-3-8B-Instruct
model_size: 8B
reward_model: ChaiML/gpt2_medium_pairwise_60m_step_937500
us_pacific_date: 2024-07-03
win_ratio: 0.48448976681166656
Resubmit model
Running pipeline stage MKMLizer
Starting job with name nousresearch-meta-llama-4941-v76-mkmlizer
Waiting for job on nousresearch-meta-llama-4941-v76-mkmlizer to finish
nousresearch-meta-llama-4941-v76-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nousresearch-meta-llama-4941-v76-mkmlizer: ║ _____ __ __ ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ /___/ ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ Version: 0.8.14 ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ https://mk1.ai ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ The license key for the current software has been verified as ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ belonging to: ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ Chai Research Corp. ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
nousresearch-meta-llama-4941-v76-mkmlizer: ║ ║
nousresearch-meta-llama-4941-v76-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nousresearch-meta-llama-4941-v76-mkmlizer: Downloaded to shared memory in 20.992s
nousresearch-meta-llama-4941-v76-mkmlizer: quantizing model to /dev/shm/model_cache
nousresearch-meta-llama-4941-v76-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nousresearch-meta-llama-4941-v76-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 4%|▍ | 13/291 [00:00<00:02, 109.24it/s] Loading 0: 8%|▊ | 24/291 [00:00<00:02, 99.30it/s] Loading 0: 13%|█▎ | 39/291 [00:00<00:02, 112.40it/s] Loading 0: 18%|█▊ | 51/291 [00:00<00:02, 111.35it/s] Loading 0: 23%|██▎ | 66/291 [00:00<00:01, 122.92it/s] Loading 0: 27%|██▋ | 79/291 [00:00<00:01, 120.92it/s] Loading 0: 32%|███▏ | 92/291 [00:01<00:03, 64.25it/s] Loading 0: 35%|███▌ | 103/291 [00:01<00:02, 71.56it/s] Loading 0: 40%|███▉ | 116/291 [00:01<00:02, 83.48it/s] Loading 0: 45%|████▍ | 130/291 [00:01<00:01, 93.62it/s] Loading 0: 49%|████▉ | 144/291 [00:01<00:01, 104.19it/s] Loading 0: 54%|█████▍ | 157/291 [00:01<00:01, 108.51it/s] Loading 0: 59%|█████▉ | 171/291 [00:01<00:01, 116.57it/s] Loading 0: 64%|██████▎ | 185/291 [00:01<00:00, 122.39it/s] Loading 0: 68%|██████▊ | 198/291 [00:02<00:01, 70.06it/s] Loading 0: 73%|███████▎ | 211/291 [00:02<00:00, 80.17it/s] Loading 0: 78%|███████▊ | 226/291 [00:02<00:00, 94.35it/s] Loading 0: 82%|████████▏ | 238/291 [00:02<00:00, 98.38it/s] Loading 0: 87%|████████▋ | 252/291 [00:02<00:00, 107.50it/s] Loading 0: 91%|█████████ | 265/291 [00:02<00:00, 110.90it/s] Loading 0: 96%|█████████▌| 279/291 [00:02<00:00, 118.20it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
nousresearch-meta-llama-4941-v76-mkmlizer: quantized model in 26.005s
nousresearch-meta-llama-4941-v76-mkmlizer: Processed model NousResearch/Meta-Llama-3-8B-Instruct in 46.998s
nousresearch-meta-llama-4941-v76-mkmlizer: creating bucket guanaco-mkml-models
nousresearch-meta-llama-4941-v76-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nousresearch-meta-llama-4941-v76-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v76
nousresearch-meta-llama-4941-v76-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v76/special_tokens_map.json
nousresearch-meta-llama-4941-v76-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v76/tokenizer_config.json
nousresearch-meta-llama-4941-v76-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v76/config.json
nousresearch-meta-llama-4941-v76-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v76/flywheel_model.0.safetensors
nousresearch-meta-llama-4941-v76-mkmlizer: loading reward model from ChaiML/gpt2_medium_pairwise_60m_step_937500
nousresearch-meta-llama-4941-v76-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:919: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v76-mkmlizer: warnings.warn(
nousresearch-meta-llama-4941-v76-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nousresearch-meta-llama-4941-v76-mkmlizer: warnings.warn(
nousresearch-meta-llama-4941-v76-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:769: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v76-mkmlizer: warnings.warn(
nousresearch-meta-llama-4941-v76-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v76-mkmlizer: warnings.warn(
Running pipeline stage MKMLizer
Starting job with name nousresearch-meta-llama-4941-v77-mkmlizer
Waiting for job on nousresearch-meta-llama-4941-v77-mkmlizer to finish
nousresearch-meta-llama-4941-v76-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
nousresearch-meta-llama-4941-v76-mkmlizer: Saving duration: 0.449s
nousresearch-meta-llama-4941-v76-mkmlizer: Processed model ChaiML/gpt2_medium_pairwise_60m_step_937500 in 7.159s
nousresearch-meta-llama-4941-v76-mkmlizer: creating bucket guanaco-reward-models
nousresearch-meta-llama-4941-v76-mkmlizer: Bucket 's3://guanaco-reward-models/' created
nousresearch-meta-llama-4941-v76-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/nousresearch-meta-llama-4941-v76_reward
nousresearch-meta-llama-4941-v76-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v76_reward/config.json
nousresearch-meta-llama-4941-v76-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v76_reward/special_tokens_map.json
nousresearch-meta-llama-4941-v76-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v76_reward/tokenizer_config.json
nousresearch-meta-llama-4941-v76-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v76_reward/vocab.json
nousresearch-meta-llama-4941-v76-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/nousresearch-meta-llama-4941-v76_reward/merges.txt
nousresearch-meta-llama-4941-v76-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v76_reward/tokenizer.json
nousresearch-meta-llama-4941-v76-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/nousresearch-meta-llama-4941-v76_reward/reward.tensors
Job nousresearch-meta-llama-4941-v76-mkmlizer completed after 73.92s with status: succeeded
Stopping job with name nousresearch-meta-llama-4941-v76-mkmlizer
Pipeline stage MKMLizer completed in 74.89s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.14s
Running pipeline stage ISVCDeployer
Creating inference service nousresearch-meta-llama-4941-v76
nousresearch-meta-llama-4941-v77-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nousresearch-meta-llama-4941-v77-mkmlizer: ║ _____ __ __ ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ /___/ ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ Version: 0.8.14 ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ https://mk1.ai ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ The license key for the current software has been verified as ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ belonging to: ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ Chai Research Corp. ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
nousresearch-meta-llama-4941-v77-mkmlizer: ║ ║
nousresearch-meta-llama-4941-v77-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Waiting for inference service nousresearch-meta-llama-4941-v76 to be ready
nousresearch-meta-llama-4941-v77-mkmlizer: Downloaded to shared memory in 26.623s
nousresearch-meta-llama-4941-v77-mkmlizer: quantizing model to /dev/shm/model_cache
nousresearch-meta-llama-4941-v77-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
nousresearch-meta-llama-4941-v77-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 4%|▍ | 12/291 [00:00<00:02, 114.50it/s] Loading 0: 8%|▊ | 24/291 [00:00<00:02, 102.08it/s] Loading 0: 13%|█▎ | 37/291 [00:00<00:02, 113.09it/s] Loading 0: 17%|█▋ | 49/291 [00:00<00:02, 103.51it/s] Loading 0: 21%|██ | 60/291 [00:00<00:02, 103.81it/s] Loading 0: 26%|██▌ | 75/291 [00:00<00:01, 112.34it/s] Loading 0: 30%|██▉ | 87/291 [00:01<00:03, 60.58it/s] Loading 0: 35%|███▌ | 102/291 [00:01<00:02, 74.35it/s] Loading 0: 38%|███▊ | 112/291 [00:01<00:02, 78.65it/s] Loading 0: 42%|████▏ | 122/291 [00:01<00:02, 82.23it/s] Loading 0: 46%|████▋ | 135/291 [00:01<00:01, 93.58it/s] Loading 0: 51%|█████ | 147/291 [00:01<00:01, 97.24it/s] Loading 0: 54%|█████▍ | 158/291 [00:01<00:01, 93.72it/s] Loading 0: 59%|█████▉ | 171/291 [00:01<00:01, 102.89it/s] Loading 0: 63%|██████▎ | 182/291 [00:01<00:01, 101.54it/s] Loading 0: 66%|██████▋ | 193/291 [00:02<00:01, 57.90it/s] Loading 0: 70%|██████▉ | 203/291 [00:02<00:01, 64.16it/s] Loading 0: 74%|███████▍ | 216/291 [00:02<00:00, 76.98it/s] Loading 0: 78%|███████▊ | 228/291 [00:02<00:00, 84.48it/s] Loading 0: 82%|████████▏ | 239/291 [00:02<00:00, 83.49it/s] Loading 0: 87%|████████▋ | 252/291 [00:02<00:00, 94.11it/s] Loading 0: 91%|█████████ | 264/291 [00:03<00:00, 98.78it/s] Loading 0: 95%|█████████▍| 275/291 [00:03<00:00, 94.42it/s] Loading 0: 99%|█████████▊| 287/291 [00:08<00:00, 6.43it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
nousresearch-meta-llama-4941-v77-mkmlizer: quantized model in 25.261s
nousresearch-meta-llama-4941-v77-mkmlizer: Processed model NousResearch/Meta-Llama-3-8B-Instruct in 51.884s
nousresearch-meta-llama-4941-v77-mkmlizer: creating bucket guanaco-mkml-models
nousresearch-meta-llama-4941-v77-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nousresearch-meta-llama-4941-v77-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v77
nousresearch-meta-llama-4941-v77-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v77/config.json
nousresearch-meta-llama-4941-v77-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v77/special_tokens_map.json
nousresearch-meta-llama-4941-v77-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v77/tokenizer_config.json
nousresearch-meta-llama-4941-v77-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v77/tokenizer.json
nousresearch-meta-llama-4941-v77-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v77/flywheel_model.0.safetensors
nousresearch-meta-llama-4941-v77-mkmlizer: loading reward model from ChaiML/gpt2_medium_pairwise_60m_step_937500
nousresearch-meta-llama-4941-v77-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:919: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v77-mkmlizer: warnings.warn(
nousresearch-meta-llama-4941-v77-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nousresearch-meta-llama-4941-v77-mkmlizer: warnings.warn(
nousresearch-meta-llama-4941-v77-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:769: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v77-mkmlizer: warnings.warn(
nousresearch-meta-llama-4941-v77-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v77-mkmlizer: warnings.warn(
nousresearch-meta-llama-4941-v77-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
nousresearch-meta-llama-4941-v77-mkmlizer: Saving duration: 0.417s
nousresearch-meta-llama-4941-v77-mkmlizer: Processed model ChaiML/gpt2_medium_pairwise_60m_step_937500 in 8.525s
nousresearch-meta-llama-4941-v77-mkmlizer: creating bucket guanaco-reward-models
nousresearch-meta-llama-4941-v77-mkmlizer: Bucket 's3://guanaco-reward-models/' created
nousresearch-meta-llama-4941-v77-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/nousresearch-meta-llama-4941-v77_reward
nousresearch-meta-llama-4941-v77-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v77_reward/config.json
nousresearch-meta-llama-4941-v77-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v77_reward/special_tokens_map.json
nousresearch-meta-llama-4941-v77-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v77_reward/vocab.json
nousresearch-meta-llama-4941-v77-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/nousresearch-meta-llama-4941-v77_reward/merges.txt
nousresearch-meta-llama-4941-v77-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v77_reward/tokenizer_config.json
nousresearch-meta-llama-4941-v77-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v77_reward/tokenizer.json
Job nousresearch-meta-llama-4941-v77-mkmlizer completed after 86.16s with status: succeeded
Stopping job with name nousresearch-meta-llama-4941-v77-mkmlizer
Pipeline stage MKMLizer completed in 86.78s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.16s
Running pipeline stage ISVCDeployer
Creating inference service nousresearch-meta-llama-4941-v77
Waiting for inference service nousresearch-meta-llama-4941-v77 to be ready
Inference service nousresearch-meta-llama-4941-v76 ready after 90.48861455917358s
Pipeline stage ISVCDeployer completed in 97.36s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.834223985671997s
Received healthy response to inference request in 1.1117067337036133s
Received healthy response to inference request in 1.1289067268371582s
Received healthy response to inference request in 1.1106581687927246s
Received healthy response to inference request in 1.1191134452819824s
5 requests
0 failed requests
5th percentile: 1.1108678817749023
10th percentile: 1.1110775947570801
20th percentile: 1.1114970207214356
30th percentile: 1.1131880760192872
40th percentile: 1.1161507606506347
50th percentile: 1.1191134452819824
60th percentile: 1.1230307579040528
70th percentile: 1.126948070526123
80th percentile: 1.2699701786041262
90th percentile: 1.5520970821380615
95th percentile: 1.693160533905029
99th percentile: 1.8060112953186034
mean time: 1.2609218120574952
Pipeline stage StressChecker completed in 8.07s
nousresearch-meta-llama_4941_v76 status is now deployed due to DeploymentManager action
nousresearch-meta-llama_4941_v76 status is now inactive due to auto deactivation removed underperforming models

Usage Metrics

Latency Metrics