nousresearch-meta-llama_4941

submission_id: nousresearch-meta-llama_4941_v67
developer_uid: end_to_end_test
formatter: {'memory_template': 'character: {bot_name} {memory}\n', 'prompt_template': '{prompt}', 'bot_template': '{bot_name}: {message}', 'user_template': '{user_name}: {message}', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 0.99, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}
model_name: nousresearch-meta-llama_4941_v67
model_repo: NousResearch/Meta-Llama-3-8B-Instruct
reward_formatter: {'bot_template': '{bot_name}: {message}', 'memory_template': 'character: {bot_name} {memory}\n', 'prompt_template': '{prompt}', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}'}
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
status: torndown
timestamp: 2024-07-01T19:25:37+00:00
Resubmit model
Running pipeline stage MKMLizer
Deleting key nousresearch-meta-llama-4941-v66_reward/tokenizer_config.json from bucket guanaco-reward-models
Starting job with name nousresearch-meta-llama-4941-v67-mkmlizer
Deleting key nousresearch-meta-llama-4941-v66_reward/vocab.json from bucket guanaco-reward-models
Waiting for job on nousresearch-meta-llama-4941-v67-mkmlizer to finish
Pipeline stage MKMLModelDeleter completed in 7.44s
nousresearch-meta-llama_4941_v66 status is now torndown due to DeploymentManager action
nousresearch-meta-llama-4941-v67-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nousresearch-meta-llama-4941-v67-mkmlizer: ║     _____            __           __                                ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║       /___/                                                         ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  Version: 0.8.14                                                    ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  https://mk1.ai                                                     ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  The license key for the current software has been verified as      ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  belonging to:                                                      ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  Chai Research Corp.                                                ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
nousresearch-meta-llama-4941-v67-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4941-v67-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nousresearch-meta-llama-4941-v67-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   5%|▍         | 14/291 [00:00<00:02, 133.94it/s]
Loading 0:  11%|█         | 32/291 [00:00<00:01, 150.53it/s]
Loading 0:  17%|█▋        | 50/291 [00:00<00:01, 155.14it/s]
Loading 0:  23%|██▎       | 68/291 [00:00<00:01, 157.36it/s]
Loading 0:  29%|██▉       | 84/291 [00:00<00:02, 84.30it/s] 
Loading 0:  35%|███▌      | 102/291 [00:00<00:01, 101.73it/s]
Loading 0:  41%|████      | 120/291 [00:01<00:01, 116.98it/s]
Loading 0:  47%|████▋     | 138/291 [00:01<00:01, 129.41it/s]
Loading 0:  54%|█████▎    | 156/291 [00:01<00:00, 136.43it/s]
Loading 0:  59%|█████▉    | 172/291 [00:01<00:00, 129.88it/s]
Loading 0:  64%|██████▍   | 187/291 [00:01<00:01, 84.80it/s] 
Loading 0:  69%|██████▉   | 202/291 [00:01<00:00, 94.50it/s]
Loading 0:  75%|███████▌  | 219/291 [00:01<00:00, 109.47it/s]
Loading 0:  80%|████████  | 233/291 [00:02<00:00, 115.58it/s]
Loading 0:  85%|████████▌ | 248/291 [00:02<00:00, 121.80it/s]
Loading 0:  91%|█████████▏| 266/291 [00:02<00:00, 133.52it/s]
Loading 0:  98%|█████████▊| 284/291 [00:02<00:00, 144.12it/s]
                                                             
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
nousresearch-meta-llama-4941-v67-mkmlizer: quantized model in 23.699s
nousresearch-meta-llama-4941-v67-mkmlizer: Processed model NousResearch/Meta-Llama-3-8B-Instruct in 45.428s
nousresearch-meta-llama-4941-v67-mkmlizer: creating bucket guanaco-mkml-models
nousresearch-meta-llama-4941-v67-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nousresearch-meta-llama-4941-v67-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v67
nousresearch-meta-llama-4941-v67-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v67/config.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v67/tokenizer_config.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v67/special_tokens_map.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v67/tokenizer.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v67/flywheel_model.0.safetensors
nousresearch-meta-llama-4941-v67-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v67-mkmlizer:   warnings.warn(
nousresearch-meta-llama-4941-v67-mkmlizer: 
Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading shards:  50%|█████     | 1/2 [00:06<00:06,  6.49s/it]
Downloading shards: 100%|██████████| 2/2 [00:08<00:00,  4.09s/it]
Downloading shards: 100%|██████████| 2/2 [00:08<00:00,  4.45s/it]
nousresearch-meta-llama-4941-v67-mkmlizer: Saving duration: 2.399s
nousresearch-meta-llama-4941-v67-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 13.413s
nousresearch-meta-llama-4941-v67-mkmlizer: creating bucket guanaco-reward-models
nousresearch-meta-llama-4941-v67-mkmlizer: Bucket 's3://guanaco-reward-models/' created
nousresearch-meta-llama-4941-v67-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/nousresearch-meta-llama-4941-v67_reward
nousresearch-meta-llama-4941-v67-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v67_reward/special_tokens_map.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v67_reward/config.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v67_reward/tokenizer_config.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v67_reward/vocab.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/nousresearch-meta-llama-4941-v67_reward/merges.txt
nousresearch-meta-llama-4941-v67-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v67_reward/tokenizer.json
nousresearch-meta-llama-4941-v67-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/nousresearch-meta-llama-4941-v67_reward/reward.tensors
Job nousresearch-meta-llama-4941-v67-mkmlizer completed after 206.68s with status: succeeded
Stopping job with name nousresearch-meta-llama-4941-v67-mkmlizer
Pipeline stage MKMLizer completed in 209.11s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.45s
Running pipeline stage ISVCDeployer
Creating inference service nousresearch-meta-llama-4941-v67
Waiting for inference service nousresearch-meta-llama-4941-v67 to be ready
Inference service nousresearch-meta-llama-4941-v67 ready after 40.7217960357666s
Pipeline stage ISVCDeployer completed in 47.31s
Running pipeline stage StressChecker
%s, retrying in %s seconds...
%s, retrying in %s seconds...
DeploymentChecksError('expected string or bytes-like object')
nousresearch-meta-llama_4941_v67 status is now failed due to DeploymentManager action
admin requested tearing down of nousresearch-meta-llama_4941_v67
Running pipeline stage ISVCDeleter
Checking if service nousresearch-meta-llama-4941-v67 is running
Tearing down inference service nousresearch-meta-llama-4941-v67
Toredown service nousresearch-meta-llama-4941-v67
Pipeline stage ISVCDeleter completed in 6.04s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key nousresearch-meta-llama-4941-v67/config.json from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4941-v67/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4941-v67/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4941-v67/tokenizer.json from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4941-v67/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key nousresearch-meta-llama-4941-v67_reward/config.json from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4941-v67_reward/merges.txt from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4941-v67_reward/reward.tensors from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4941-v67_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4941-v67_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4941-v67_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key nousresearch-meta-llama-4941-v67_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 2.43s
nousresearch-meta-llama_4941_v67 status is now torndown due to DeploymentManager action