submission_id: pjmixers-llama-3-curseds_4945_v2
developer_uid: xzuyn
formatter: {'memory_template': "<|start_header_id|>system<|end_header_id|>\\n\\n{bot_name}'s Persona: {memory}<|eot_id|>", 'prompt_template': '\\n\\nExample Messages:\\n{prompt}', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\\n\\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\\n\\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\\n\\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.25, 'top_p': 1.0, 'min_p': 0.1, 'top_k': 50, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|eot_id|>', '<|end_of_text|>'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64, 'reward_max_token_input': 512}
model_name: pjmixers-llama-3-curseds_4945_v2
model_repo: PJMixers/LLaMa-3-CursedStock-v2.0-8B
reward_formatter: {'bot_template': '{bot_name}: {message}\\n', 'memory_template': "{bot_name}'s Persona: {memory}\\n####\\n", 'prompt_template': '{prompt}\\n<START>\\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\\n'}
reward_repo: Jellywibble/gpt2_xl_pairwise_89m_step_347634
status: torndown
timestamp: 2024-07-27T01:00:41+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name pjmixers-llama-3-curseds-4945-v2-mkmlizer
Waiting for job on pjmixers-llama-3-curseds-4945-v2-mkmlizer to finish
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ _____ __ __ ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ /___/ ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ Version: 0.9.7 ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ https://mk1.ai ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ The license key for the current software has been verified as ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ belonging to: ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ Chai Research Corp. ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ║ ║
pjmixers-llama-3-curseds-4945-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Downloaded to shared memory in 42.580s
pjmixers-llama-3-curseds-4945-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmparhptfjl, device:0
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
pjmixers-llama-3-curseds-4945-v2-mkmlizer: quantized model in 28.191s
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Processed model PJMixers/LLaMa-3-CursedStock-v2.0-8B in 70.771s
pjmixers-llama-3-curseds-4945-v2-mkmlizer: creating bucket guanaco-mkml-models
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
pjmixers-llama-3-curseds-4945-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/pjmixers-llama-3-curseds-4945-v2
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/pjmixers-llama-3-curseds-4945-v2/config.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/pjmixers-llama-3-curseds-4945-v2/special_tokens_map.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/pjmixers-llama-3-curseds-4945-v2/tokenizer_config.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/pjmixers-llama-3-curseds-4945-v2/tokenizer.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/pjmixers-llama-3-curseds-4945-v2/flywheel_model.0.safetensors
pjmixers-llama-3-curseds-4945-v2-mkmlizer: loading reward model from Jellywibble/gpt2_xl_pairwise_89m_step_347634
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 1%| | 2/291 [00:04<11:26, 2.38s/it] Loading 0: 2%|▏ | 6/291 [00:04<03:05, 1.54it/s] Loading 0: 3%|▎ | 10/291 [00:05<01:38, 2.85it/s] Loading 0: 5%|▍ | 14/291 [00:05<00:59, 4.65it/s] Loading 0: 7%|▋ | 19/291 [00:05<00:35, 7.59it/s] Loading 0: 8%|▊ | 23/291 [00:05<00:26, 10.19it/s] Loading 0: 10%|▉ | 28/291 [00:05<00:18, 14.35it/s] Loading 0: 11%|█ | 32/291 [00:05<00:14, 17.27it/s] Loading 0: 12%|█▏ | 36/291 [00:05<00:12, 20.45it/s] Loading 0: 14%|█▎ | 40/291 [00:05<00:10, 23.82it/s] Loading 0: 15%|█▌ | 44/291 [00:06<00:09, 25.77it/s] Loading 0: 17%|█▋ | 49/291 [00:06<00:07, 30.73it/s] Loading 0: 18%|█▊ | 53/291 [00:06<00:07, 31.30it/s] Loading 0: 20%|█▉ | 58/291 [00:06<00:06, 35.58it/s] Loading 0: 22%|██▏ | 63/291 [00:06<00:09, 24.67it/s] Loading 0: 23%|██▎ | 67/291 [00:06<00:08, 27.40it/s] Loading 0: 24%|██▍ | 71/291 [00:07<00:07, 28.57it/s] Loading 0: 26%|██▌ | 76/291 [00:07<00:06, 32.79it/s] Loading 0: 27%|██▋ | 80/291 [00:07<00:06, 32.44it/s] Loading 0: 29%|██▉ | 85/291 [00:07<00:05, 36.34it/s] Loading 0: 31%|███ | 89/291 [00:07<00:05, 35.00it/s] Loading 0: 32%|███▏ | 94/291 [00:07<00:05, 38.04it/s] Loading 0: 34%|███▎ | 98/291 [00:07<00:05, 35.93it/s] Loading 0: 35%|███▌ | 103/291 [00:07<00:04, 38.75it/s] Loading 0: 37%|███▋ | 108/291 [00:07<00:04, 36.94it/s] Loading 0: 38%|███▊ | 112/291 [00:08<00:04, 37.42it/s] Loading 0: 40%|███▉ | 116/291 [00:08<00:07, 24.10it/s] Loading 0: 42%|████▏ | 121/291 [00:08<00:06, 28.20it/s] Loading 0: 43%|████▎ | 125/291 [00:08<00:05, 28.96it/s] Loading 0: 45%|████▍ | 130/291 [00:08<00:04, 32.78it/s] Loading 0: 46%|████▌ | 134/291 [00:08<00:04, 31.91it/s] Loading 0: 48%|████▊ | 139/291 [00:08<00:04, 35.31it/s] Loading 0: 49%|████▉ | 143/291 [00:09<00:04, 34.27it/s] Loading 0: 51%|█████ | 148/291 [00:09<00:03, 36.93it/s] Loading 0: 52%|█████▏ | 152/291 [00:09<00:04, 34.18it/s] Loading 0: 54%|█████▍ | 157/291 [00:09<00:03, 37.13it/s] Loading 0: 55%|█████▌ | 161/291 [00:09<00:03, 35.71it/s] Loading 0: 57%|█████▋ | 166/291 [00:09<00:04, 27.05it/s] Loading 0: 58%|█████▊ | 170/291 [00:10<00:04, 28.20it/s] Loading 0: 60%|██████ | 175/291 [00:10<00:03, 31.73it/s] Loading 0: 62%|██████▏ | 179/291 [00:10<00:03, 31.88it/s] Loading 0: 63%|██████▎ | 184/291 [00:10<00:03, 35.25it/s] Loading 0: 65%|██████▍ | 188/291 [00:10<00:03, 34.13it/s] Loading 0: 66%|██████▋ | 193/291 [00:10<00:02, 36.68it/s] Loading 0: 68%|██████▊ | 197/291 [00:10<00:02, 34.54it/s] Loading 0: 69%|██████▉ | 202/291 [00:10<00:02, 37.47it/s] Loading 0: 71%|███████ | 206/291 [00:10<00:02, 35.95it/s] Loading 0: 73%|███████▎ | 211/291 [00:11<00:02, 38.17it/s] Loading 0: 74%|███████▍ | 215/291 [00:11<00:02, 25.42it/s] Loading 0: 76%|███████▌ | 220/291 [00:11<00:02, 29.66it/s] Loading 0: 77%|███████▋ | 224/291 [00:11<00:02, 30.29it/s] Loading 0: 79%|███████▊ | 229/291 [00:11<00:01, 33.75it/s] Loading 0: 80%|████████ | 233/291 [00:11<00:01, 33.41it/s] Loading 0: 82%|████████▏ | 238/291 [00:11<00:01, 36.23it/s] Loading 0: 83%|████████▎ | 242/291 [00:12<00:01, 35.01it/s] Loading 0: 85%|████████▍ | 247/291 [00:12<00:01, 37.78it/s] Loading 0: 86%|████████▋ | 251/291 [00:12<00:01, 36.05it/s] Loading 0: 88%|████████▊ | 256/291 [00:12<00:00, 38.72it/s] Loading 0: 89%|████████▉ | 260/291 [00:12<00:00, 35.82it/s] Loading 0: 91%|█████████ | 265/291 [00:12<00:00, 37.48it/s] Loading 0: 92%|█████████▏| 269/291 [00:12<00:00, 25.02it/s] Loading 0: 94%|█████████▍| 274/291 [00:13<00:00, 29.34it/s] Loading 0: 96%|█████████▌| 278/291 [00:13<00:00, 30.15it/s] Loading 0: 97%|█████████▋| 283/291 [00:13<00:00, 33.59it/s] Loading 0: 99%|█████████▊| 287/291 [00:13<00:00, 33.18it/s] /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
pjmixers-llama-3-curseds-4945-v2-mkmlizer: warnings.warn(
pjmixers-llama-3-curseds-4945-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
pjmixers-llama-3-curseds-4945-v2-mkmlizer: warnings.warn(
pjmixers-llama-3-curseds-4945-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
pjmixers-llama-3-curseds-4945-v2-mkmlizer: warnings.warn(
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Saving duration: 1.365s
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Processed model Jellywibble/gpt2_xl_pairwise_89m_step_347634 in 10.800s
pjmixers-llama-3-curseds-4945-v2-mkmlizer: creating bucket guanaco-reward-models
pjmixers-llama-3-curseds-4945-v2-mkmlizer: Bucket 's3://guanaco-reward-models/' created
pjmixers-llama-3-curseds-4945-v2-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/pjmixers-llama-3-curseds-4945-v2_reward
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/pjmixers-llama-3-curseds-4945-v2_reward/config.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/pjmixers-llama-3-curseds-4945-v2_reward/special_tokens_map.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/pjmixers-llama-3-curseds-4945-v2_reward/tokenizer_config.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/pjmixers-llama-3-curseds-4945-v2_reward/merges.txt
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/pjmixers-llama-3-curseds-4945-v2_reward/vocab.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/pjmixers-llama-3-curseds-4945-v2_reward/tokenizer.json
pjmixers-llama-3-curseds-4945-v2-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/pjmixers-llama-3-curseds-4945-v2_reward/reward.tensors
Job pjmixers-llama-3-curseds-4945-v2-mkmlizer completed after 117.07s with status: succeeded
Stopping job with name pjmixers-llama-3-curseds-4945-v2-mkmlizer
Pipeline stage MKMLizer completed in 118.08s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service pjmixers-llama-3-curseds-4945-v2
Waiting for inference service pjmixers-llama-3-curseds-4945-v2 to be ready
Inference service pjmixers-llama-3-curseds-4945-v2 ready after 81.18523788452148s
Pipeline stage ISVCDeployer completed in 82.88s
Running pipeline stage StressChecker
%s, retrying in %s seconds...
%s, retrying in %s seconds...
DeploymentChecksError("expected string or bytes-like object, got 'AssertionError'")
pjmixers-llama-3-curseds_4945_v2 status is now failed due to DeploymentManager action
admin requested tearing down of pjmixers-llama-3-curseds_4945_v2
Running pipeline stage ISVCDeleter
Checking if service pjmixers-llama-3-curseds-4945-v2 is running
Tearing down inference service pjmixers-llama-3-curseds-4945-v2
Toredown service pjmixers-llama-3-curseds-4945-v2
Pipeline stage ISVCDeleter completed in 2.99s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key pjmixers-llama-3-curseds-4945-v2/config.json from bucket guanaco-mkml-models
Deleting key pjmixers-llama-3-curseds-4945-v2/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key pjmixers-llama-3-curseds-4945-v2/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key pjmixers-llama-3-curseds-4945-v2/tokenizer.json from bucket guanaco-mkml-models
Deleting key pjmixers-llama-3-curseds-4945-v2/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key pjmixers-llama-3-curseds-4945-v2_reward/config.json from bucket guanaco-reward-models
Deleting key pjmixers-llama-3-curseds-4945-v2_reward/merges.txt from bucket guanaco-reward-models
Deleting key pjmixers-llama-3-curseds-4945-v2_reward/reward.tensors from bucket guanaco-reward-models
Deleting key pjmixers-llama-3-curseds-4945-v2_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key pjmixers-llama-3-curseds-4945-v2_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key pjmixers-llama-3-curseds-4945-v2_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key pjmixers-llama-3-curseds-4945-v2_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 2.80s
pjmixers-llama-3-curseds_4945_v2 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics