submission_id: undi95-meta-llama-3-70b_6209_v29
developer_uid: Jellywibble
best_of: 2
display_name: meta-llama-3-70b-1500ctx
family_friendly_score: 0.0
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s Persona: {memory}<|im_end|>\n", 'prompt_template': '<|im_start|>system\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 50, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|im_end|>', '<|im_start|>', '\n\n'], 'max_input_tokens': 1500, 'best_of': 2, 'max_output_tokens': 64, 'reward_max_token_input': 256}
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: Undi95/Meta-Llama-3-70B-Instruct-hf
max_input_tokens: 1500
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: Undi95/Meta-Llama-3-70B-
model_name: meta-llama-3-70b-1500ctx
model_num_parameters: 70553706496.0
model_repo: Undi95/Meta-Llama-3-70B-Instruct-hf
model_size: 71B
num_battles: 76
num_wins: 34
ranking_group: single
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "''", 'prompt_template': "''", 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
status: torndown
submission_type: basic
timestamp: 2024-07-25T15:47:25+00:00
us_pacific_date: 2024-07-25
win_ratio: 0.4473684210526316
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name undi95-meta-llama-3-70b-6209-v29-mkmlizer
Waiting for job on undi95-meta-llama-3-70b-6209-v29-mkmlizer to finish
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-noromaid-v0_8068_v141: ('http://neversleep-noromaid-v0-8068-v141-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║     _____            __           __                                ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║       /___/                                                         ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║                                                                     ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  Version: 0.9.7                                                     ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  https://mk1.ai                                                     ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║                                                                     ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  The license key for the current software has been verified as      ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  belonging to:                                                      ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║                                                                     ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  Chai Research Corp.                                                ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ║                                                                     ║
undi95-meta-llama-3-70b-6209-v29-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
undi95-meta-llama-3-70b-6209-v29-mkmlizer: Downloaded to shared memory in 288.259s
undi95-meta-llama-3-70b-6209-v29-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpi6840rc2, device:0
undi95-meta-llama-3-70b-6209-v29-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Retrying (%r) after connection broken by '%r': %s
undi95-meta-llama-3-70b-6209-v29-mkmlizer: quantized model in 142.012s
undi95-meta-llama-3-70b-6209-v29-mkmlizer: Processed model Undi95/Meta-Llama-3-70B-Instruct-hf in 430.271s
undi95-meta-llama-3-70b-6209-v29-mkmlizer: creating bucket guanaco-mkml-models
undi95-meta-llama-3-70b-6209-v29-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
undi95-meta-llama-3-70b-6209-v29-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/config.json
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/special_tokens_map.json
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/tokenizer_config.json
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/tokenizer.json
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.5.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/flywheel_model.5.safetensors
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/flywheel_model.2.safetensors
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/flywheel_model.3.safetensors
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/flywheel_model.0.safetensors
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/flywheel_model.1.safetensors
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.4.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v29/flywheel_model.4.safetensors
undi95-meta-llama-3-70b-6209-v29-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
undi95-meta-llama-3-70b-6209-v29-mkmlizer: 
Loading 0:   0%|          | 0/723 [00:00<?, ?it/s]
Loading 0:   0%|          | 3/723 [00:00<01:24,  8.50it/s]
Loading 0:   1%|          | 4/723 [00:00<02:07,  5.63it/s]
Loading 0:   1%|          | 5/723 [00:01<02:53,  4.14it/s]
Loading 0:   1%|          | 8/723 [00:01<01:27,  8.14it/s]
Loading 0:   1%|▏         | 10/723 [00:01<01:16,  9.28it/s]
Loading 0:   2%|▏         | 12/723 [00:01<01:28,  8.05it/s]
Loading 0:   2%|▏         | 14/723 [00:01<01:18,  9.07it/s]
Loading 0:   2%|▏         | 16/723 [00:01<01:14,  9.52it/s]
Loading 0:   2%|▏         | 18/723 [00:02<01:53,  6.22it/s]
Loading 0:   3%|▎         | 21/723 [00:02<01:25,  8.23it/s]
Loading 0:   3%|▎         | 23/723 [00:03<01:42,  6.82it/s]
Loading 0:   4%|▎         | 27/723 [00:03<01:06, 10.53it/s]
Loading 0:   4%|▍         | 30/723 [00:03<01:03, 10.94it/s]
Loading 0:   4%|▍         | 32/723 [00:03<01:22,  8.40it/s]
Loading 0:   5%|▍         | 36/723 [00:04<00:57, 12.04it/s]
Loading 0:   5%|▌         | 39/723 [00:04<00:48, 14.15it/s]
Loading 0:   6%|▌         | 42/723 [00:04<00:49, 13.84it/s]
Loading 0:   6%|▌         | 44/723 [00:04<01:06, 10.26it/s]
Loading 0:   6%|▋         | 46/723 [00:05<01:09,  9.73it/s]
Loading 0:   7%|▋         | 48/723 [00:05<01:07,  9.96it/s]
Loading 0:   7%|▋         | 50/723 [00:05<01:25,  7.83it/s]
Loading 0:   7%|▋         | 54/723 [00:05<00:56, 11.75it/s]
Loading 0:   8%|▊         | 57/723 [00:05<00:56, 11.81it/s]
Loading 0:   8%|▊         | 59/723 [00:06<01:15,  8.78it/s]
Loading 0:   9%|▊         | 63/723 [00:06<00:52, 12.49it/s]
Loading 0:   9%|▉         | 66/723 [00:06<00:45, 14.44it/s]
Loading 0:  10%|▉         | 69/723 [00:06<00:53, 12.31it/s]
Loading 0:  10%|▉         | 71/723 [00:07<01:01, 10.61it/s]
Loading 0:  10%|█         | 75/723 [00:07<00:43, 14.75it/s]
Loading 0:  11%|█         | 78/723 [00:07<00:43, 14.92it/s]
Loading 0:  11%|█▏        | 82/723 [00:07<00:34, 18.40it/s]
Loading 0:  12%|█▏        | 85/723 [00:07<00:33, 19.20it/s]
Loading 0:  12%|█▏        | 88/723 [00:07<00:32, 19.76it/s]
Loading 0:  13%|█▎        | 91/723 [00:08<00:29, 21.19it/s]
Loading 0:  13%|█▎        | 94/723 [00:08<00:39, 15.92it/s]
Loading 0:  13%|█▎        | 96/723 [00:08<00:39, 15.86it/s]
Loading 0:  14%|█▍        | 100/723 [00:08<00:31, 19.90it/s]
Loading 0:  14%|█▍        | 103/723 [00:08<00:30, 20.21it/s]
Loading 0:  15%|█▍        | 106/723 [00:08<00:30, 20.39it/s]
Loading 0:  15%|█▌        | 109/723 [00:09<00:28, 21.80it/s]
Loading 0:  15%|█▌        | 112/723 [00:09<00:33, 18.37it/s]
Loading 0:  16%|█▌        | 115/723 [00:09<00:29, 20.35it/s]
Loading 0:  16%|█▋        | 118/723 [00:09<00:34, 17.43it/s]
Loading 0:  17%|█▋        | 121/723 [00:09<00:32, 18.68it/s]
Loading 0:  17%|█▋        | 124/723 [00:09<00:30, 19.67it/s]
Loading 0:  17%|█▋        | 124/723 [00:20<00:30, 19.67it/s]
Loading 0:  17%|█▋        | 125/723 [00:25<19:20,  1.94s/it]
Loading 0:  18%|█▊        | 127/723 [00:25<14:21,  1.45s/it]
Loading 0:  18%|█▊        | 130/723 [00:25<09:31,  1.04it/s]
Loading 0:  18%|█▊        | 132/723 [00:25<07:13,  1.36it/s]
Loading 0:  19%|█▉        | 136/723 [00:25<04:15,  2.30it/s]
Loading 0:  19%|█▉        | 140/723 [00:25<02:44,  3.55it/s]
Loading 0:  20%|█▉        | 143/723 [00:26<02:12,  4.38it/s]
Loading 0:  20%|██        | 146/723 [00:26<01:40,  5.73it/s]
Loading 0:  21%|██        | 149/723 [00:26<01:25,  6.69it/s]
Loading 0:  22%|██▏       | 156/723 [00:26<00:49, 11.43it/s]
Loading 0:  22%|██▏       | 159/723 [00:26<00:46, 12.18it/s]
Loading 0:  23%|██▎       | 163/723 [00:27<00:36, 15.17it/s]
Loading 0:  23%|██▎       | 168/723 [00:27<00:33, 16.38it/s]
Loading 0:  24%|██▎       | 171/723 [00:27<00:36, 14.96it/s]
Loading 0:  24%|██▍       | 175/723 [00:27<00:31, 17.30it/s]
Loading 0:  25%|██▍       | 178/723 [00:27<00:30, 18.03it/s]
Loading 0:  25%|██▌       | 181/723 [00:27<00:27, 19.54it/s]
Loading 0:  25%|██▌       | 184/723 [00:28<00:26, 20.09it/s]
Loading 0:  26%|██▌       | 187/723 [00:28<00:26, 20.46it/s]
Loading 0:  26%|██▋       | 190/723 [00:28<00:24, 22.04it/s]
Loading 0:  27%|██▋       | 194/723 [00:28<00:27, 19.10it/s]
Loading 0:  27%|██▋       | 197/723 [00:28<00:32, 16.24it/s]
Loading 0:  28%|██▊       | 201/723 [00:28<00:25, 20.28it/s]
Loading 0:  28%|██▊       | 204/723 [00:29<00:28, 18.39it/s]
Loading 0:  29%|██▉       | 208/723 [00:29<00:24, 21.37it/s]
Loading 0:  29%|██▉       | 211/723 [00:29<00:23, 21.38it/s]
Loading 0:  30%|██▉       | 214/723 [00:29<00:23, 21.54it/s]
Loading 0:  30%|███       | 217/723 [00:29<00:22, 22.92it/s]
Loading 0:  30%|███       | 220/723 [00:29<00:30, 16.67it/s]
Loading 0:  31%|███       | 223/723 [00:30<00:27, 18.02it/s]
Loading 0:  31%|███▏      | 226/723 [00:30<00:24, 20.14it/s]
Loading 0:  32%|███▏      | 229/723 [00:30<00:23, 20.84it/s]
Loading 0:  32%|███▏      | 232/723 [00:30<00:23, 21.16it/s]
Loading 0:  33%|███▎      | 235/723 [00:30<00:21, 22.64it/s]
Loading 0:  33%|███▎      | 238/723 [00:30<00:25, 19.08it/s]
Loading 0:  33%|███▎      | 241/723 [00:30<00:23, 20.56it/s]
Loading 0:  34%|███▎      | 244/723 [00:31<00:27, 17.51it/s]
Loading 0:  34%|███▍      | 247/723 [00:31<00:25, 18.78it/s]
Loading 0:  35%|███▍      | 250/723 [00:31<00:23, 19.73it/s]
Loading 0:  35%|███▍      | 253/723 [00:31<00:21, 21.58it/s]
Loading 0:  35%|███▌      | 256/723 [00:31<00:21, 21.66it/s]
Loading 0:  36%|███▌      | 259/723 [00:31<00:21, 21.61it/s]
Loading 0:  36%|███▌      | 262/723 [00:31<00:19, 23.11it/s]
Loading 0:  37%|███▋      | 266/723 [00:32<00:17, 26.56it/s]
Loading 0:  37%|███▋      | 269/723 [00:47<10:49,  1.43s/it]
Loading 0:  37%|███▋      | 270/723 [00:47<09:39,  1.28s/it]
Loading 0:  38%|███▊      | 273/723 [00:47<06:35,  1.14it/s]
Loading 0:  38%|███▊      | 275/723 [00:47<05:12,  1.43it/s]
Loading 0:  39%|███▊      | 280/723 [00:47<02:51,  2.59it/s]
Loading 0:  39%|███▉      | 283/723 [00:47<02:07,  3.44it/s]
Loading 0:  40%|███▉      | 286/723 [00:48<01:35,  4.55it/s]
Loading 0:  40%|███▉      | 289/723 [00:48<01:12,  6.01it/s]
Loading 0:  41%|████      | 293/723 [00:48<00:50,  8.47it/s]
Loading 0:  41%|████      | 296/723 [00:48<00:48,  8.86it/s]
Loading 0:  41%|████▏     | 299/723 [00:48<00:39, 10.73it/s]
Loading 0:  42%|████▏     | 302/723 [00:49<00:39, 10.79it/s]
Loading 0:  42%|████▏     | 307/723 [00:49<00:27, 15.21it/s]
Loading 0:  43%|████▎     | 310/723 [00:49<00:25, 16.19it/s]
Loading 0:  43%|████▎     | 313/723 [00:49<00:23, 17.50it/s]
Loading 0:  44%|████▎     | 316/723 [00:49<00:21, 19.37it/s]
Loading 0:  44%|████▍     | 319/723 [00:49<00:18, 21.27it/s]
Loading 0:  45%|████▍     | 322/723 [00:49<00:25, 16.04it/s]
Loading 0:  45%|████▍     | 325/723 [00:50<00:23, 16.65it/s]
Loading 0:  45%|████▌     | 328/723 [00:50<00:21, 18.12it/s]
Loading 0:  46%|████▌     | 331/723 [00:50<00:20, 19.21it/s]
Loading 0:  46%|████▌     | 334/723 [00:50<00:18, 20.63it/s]
Loading 0:  47%|████▋     | 337/723 [00:50<00:18, 20.33it/s]
Loading 0:  47%|████▋     | 340/723 [00:50<00:18, 21.02it/s]
Loading 0:  47%|████▋     | 343/723 [00:50<00:16, 22.49it/s]
Loading 0:  48%|████▊     | 346/723 [00:51<00:23, 16.17it/s]
Loading 0:  48%|████▊     | 348/723 [00:51<00:23, 15.84it/s]
Loading 0:  49%|████▊     | 352/723 [00:51<00:18, 19.90it/s]
Loading 0:  49%|████▉     | 355/723 [00:51<00:18, 19.76it/s]
Loading 0:  50%|████▉     | 358/723 [00:51<00:17, 20.44it/s]
Loading 0:  50%|████▉     | 361/723 [00:51<00:16, 21.89it/s]
Loading 0:  50%|█████     | 364/723 [00:52<00:19, 18.61it/s]
Loading 0:  51%|█████     | 367/723 [00:52<00:17, 20.65it/s]
Loading 0:  51%|█████     | 370/723 [00:52<00:20, 17.49it/s]
Loading 0:  52%|█████▏    | 373/723 [00:52<00:18, 18.69it/s]
Loading 0:  52%|█████▏    | 376/723 [00:52<00:17, 19.74it/s]
Loading 0:  52%|█████▏    | 379/723 [00:52<00:15, 21.58it/s]
Loading 0:  53%|█████▎    | 382/723 [00:52<00:16, 20.93it/s]
Loading 0:  53%|█████▎    | 385/723 [00:53<00:15, 21.43it/s]
Loading 0:  54%|█████▎    | 388/723 [00:53<00:14, 22.85it/s]
Loading 0:  54%|█████▍    | 391/723 [00:53<00:13, 24.10it/s]
Loading 0:  54%|█████▍    | 394/723 [00:53<00:17, 18.98it/s]
Loading 0:  55%|█████▍    | 397/723 [00:53<00:18, 17.78it/s]
Loading 0:  55%|█████▌    | 400/723 [00:53<00:17, 18.46it/s]
Loading 0:  56%|█████▌    | 402/723 [01:08<08:55,  1.67s/it]
Loading 0:  56%|█████▌    | 405/723 [01:08<06:05,  1.15s/it]
Loading 0:  56%|█████▋    | 408/723 [01:08<04:14,  1.24it/s]
Loading 0:  57%|█████▋    | 410/723 [01:09<03:21,  1.55it/s]
Loading 0:  57%|█████▋    | 415/723 [01:09<01:52,  2.74it/s]
Loading 0:  58%|█████▊    | 419/723 [01:09<01:16,  3.99it/s]
Loading 0:  58%|█████▊    | 422/723 [01:09<01:03,  4.77it/s]
Loading 0:  59%|█████▉    | 425/723 [01:09<00:48,  6.11it/s]
Loading 0:  59%|█████▉    | 428/723 [01:10<00:42,  6.97it/s]
Loading 0:  60%|█████▉    | 433/723 [01:10<00:27, 10.45it/s]
Loading 0:  60%|██████    | 436/723 [01:10<00:24, 11.91it/s]
Loading 0:  61%|██████    | 439/723 [01:10<00:20, 13.58it/s]
Loading 0:  61%|██████    | 442/723 [01:10<00:17, 15.61it/s]
Loading 0:  62%|██████▏   | 445/723 [01:10<00:15, 18.01it/s]
Loading 0:  62%|██████▏   | 448/723 [01:11<00:18, 14.64it/s]
Loading 0:  62%|██████▏   | 451/723 [01:11<00:17, 15.31it/s]
Loading 0:  63%|██████▎   | 454/723 [01:11<00:15, 16.87it/s]
Loading 0:  63%|██████▎   | 457/723 [01:11<00:14, 17.93it/s]
Loading 0:  64%|██████▎   | 460/723 [01:11<00:13, 19.60it/s]
Loading 0:  64%|██████▍   | 463/723 [01:11<00:13, 19.60it/s]
Loading 0:  64%|██████▍   | 466/723 [01:11<00:12, 20.33it/s]
Loading 0:  65%|██████▍   | 469/723 [01:11<00:11, 22.16it/s]
Loading 0:  65%|██████▌   | 472/723 [01:12<00:15, 16.29it/s]
Loading 0:  66%|██████▌   | 474/723 [01:12<00:15, 16.23it/s]
Loading 0:  66%|██████▌   | 478/723 [01:12<00:12, 19.73it/s]
Loading 0:  67%|██████▋   | 481/723 [01:12<00:12, 19.71it/s]
Loading 0:  67%|██████▋   | 484/723 [01:12<00:11, 20.32it/s]
Loading 0:  67%|██████▋   | 487/723 [01:12<00:10, 21.91it/s]
Loading 0:  68%|██████▊   | 490/723 [01:13<00:12, 18.42it/s]
Loading 0:  68%|██████▊   | 493/723 [01:13<00:11, 20.26it/s]
Loading 0:  69%|██████▊   | 496/723 [01:13<00:13, 17.28it/s]
Loading 0:  69%|██████▉   | 499/723 [01:13<00:12, 18.59it/s]
Loading 0:  69%|██████▉   | 502/723 [01:13<00:11, 19.02it/s]
Loading 0:  70%|██████▉   | 505/723 [01:13<00:10, 20.68it/s]
Loading 0:  70%|███████   | 508/723 [01:14<00:10, 20.36it/s]
Loading 0:  71%|███████   | 511/723 [01:14<00:10, 20.81it/s]
Loading 0:  71%|███████   | 514/723 [01:14<00:09, 22.10it/s]
Loading 0:  72%|███████▏  | 517/723 [01:14<00:08, 23.61it/s]
Loading 0:  72%|███████▏  | 520/723 [01:14<00:10, 18.56it/s]
Loading 0:  72%|███████▏  | 523/723 [01:14<00:11, 17.46it/s]
Loading 0:  73%|███████▎  | 526/723 [01:15<00:10, 18.19it/s]
Loading 0:  73%|███████▎  | 528/723 [01:15<00:11, 17.61it/s]
Loading 0:  74%|███████▎  | 532/723 [01:15<00:09, 21.09it/s]
Loading 0:  74%|███████▍  | 535/723 [01:15<00:09, 20.65it/s]
Loading 0:  74%|███████▍  | 538/723 [01:29<04:30,  1.46s/it]
Loading 0:  75%|███████▍  | 540/723 [01:30<03:31,  1.16s/it]
Loading 0:  75%|███████▌  | 543/723 [01:30<02:24,  1.25it/s]
Loading 0:  76%|███████▌  | 546/723 [01:30<01:41,  1.74it/s]
Loading 0:  76%|███████▌  | 548/723 [01:30<01:20,  2.18it/s]
Loading 0:  76%|███████▌  | 550/723 [01:30<01:02,  2.78it/s]
Loading 0:  76%|███████▋  | 553/723 [01:30<00:42,  3.96it/s]
Loading 0:  77%|███████▋  | 555/723 [01:30<00:34,  4.85it/s]
Loading 0:  77%|███████▋  | 559/723 [01:31<00:21,  7.48it/s]
Loading 0:  78%|███████▊  | 562/723 [01:31<00:17,  9.24it/s]
Loading 0:  78%|███████▊  | 565/723 [01:31<00:14, 11.28it/s]
Loading 0:  79%|███████▊  | 568/723 [01:31<00:11, 13.76it/s]
Loading 0:  79%|███████▉  | 571/723 [01:31<00:09, 16.44it/s]
Loading 0:  79%|███████▉  | 574/723 [01:31<00:10, 13.86it/s]
Loading 0:  80%|███████▉  | 577/723 [01:32<00:09, 14.68it/s]
Loading 0:  80%|████████  | 580/723 [01:32<00:08, 16.51it/s]
Loading 0:  81%|████████  | 583/723 [01:32<00:07, 17.95it/s]
Loading 0:  81%|████████  | 586/723 [01:32<00:06, 19.71it/s]
Loading 0:  81%|████████▏ | 589/723 [01:32<00:06, 19.57it/s]
Loading 0:  82%|████████▏ | 592/723 [01:32<00:06, 20.55it/s]
Loading 0:  82%|████████▏ | 595/723 [01:32<00:05, 22.34it/s]
Loading 0:  83%|████████▎ | 598/723 [01:33<00:07, 16.27it/s]
Loading 0:  83%|████████▎ | 600/723 [01:33<00:07, 16.11it/s]
Loading 0:  84%|████████▎ | 604/723 [01:33<00:06, 19.72it/s]
Loading 0:  84%|████████▍ | 607/723 [01:33<00:05, 19.84it/s]
Loading 0:  84%|████████▍ | 610/723 [01:33<00:05, 20.60it/s]
Loading 0:  85%|████████▍ | 613/723 [01:33<00:04, 22.15it/s]
Loading 0:  85%|████████▌ | 616/723 [01:34<00:05, 18.75it/s]
Loading 0:  86%|████████▌ | 619/723 [01:34<00:05, 20.62it/s]
Loading 0:  86%|████████▌ | 622/723 [01:34<00:05, 17.64it/s]
Loading 0:  86%|████████▋ | 625/723 [01:34<00:05, 18.98it/s]
Loading 0:  87%|████████▋ | 628/723 [01:34<00:04, 19.44it/s]
Loading 0:  87%|████████▋ | 631/723 [01:34<00:04, 21.02it/s]
Loading 0:  88%|████████▊ | 634/723 [01:34<00:04, 20.89it/s]
Loading 0:  88%|████████▊ | 637/723 [01:35<00:04, 21.25it/s]
Loading 0:  89%|████████▊ | 640/723 [01:35<00:03, 22.52it/s]
Loading 0:  89%|████████▉ | 643/723 [01:35<00:03, 23.88it/s]
Loading 0:  89%|████████▉ | 646/723 [01:35<00:04, 18.59it/s]
Loading 0:  90%|████████▉ | 649/723 [01:35<00:04, 17.50it/s]
Loading 0:  90%|█████████ | 652/723 [01:35<00:03, 18.20it/s]
Loading 0:  90%|█████████ | 654/723 [01:35<00:04, 17.15it/s]
Loading 0:  91%|█████████ | 658/723 [01:36<00:03, 20.52it/s]
Loading 0:  91%|█████████▏| 661/723 [01:36<00:03, 20.27it/s]
Loading 0:  92%|█████████▏| 664/723 [01:36<00:02, 20.69it/s]
Loading 0:  92%|█████████▏| 667/723 [01:36<00:02, 21.87it/s]
Loading 0:  93%|█████████▎| 671/723 [01:36<00:02, 24.90it/s]
Loading 0:  93%|█████████▎| 674/723 [01:36<00:02, 17.77it/s]
Loading 0:  93%|█████████▎| 674/723 [01:51<00:02, 17.77it/s]
Loading 0:  93%|█████████▎| 675/723 [01:51<01:23,  1.73s/it]
Loading 0:  94%|█████████▍| 678/723 [01:51<00:53,  1.18s/it]
Loading 0:  94%|█████████▍| 680/723 [01:51<00:39,  1.08it/s]
Loading 0:  95%|█████████▍| 685/723 [01:51<00:19,  1.96it/s]
Loading 0:  95%|█████████▌| 688/723 [01:52<00:13,  2.63it/s]
Loading 0:  96%|█████████▌| 691/723 [01:52<00:09,  3.53it/s]
Loading 0:  96%|█████████▌| 694/723 [01:52<00:06,  4.75it/s]
Loading 0:  96%|█████████▋| 697/723 [01:52<00:04,  6.30it/s]
Loading 0:  97%|█████████▋| 700/723 [01:52<00:03,  7.09it/s]
Loading 0:  97%|█████████▋| 703/723 [01:52<00:02,  8.67it/s]
Loading 0:  98%|█████████▊| 706/723 [01:53<00:01, 10.65it/s]
Loading 0:  98%|█████████▊| 709/723 [01:53<00:01, 12.68it/s]
Loading 0:  98%|█████████▊| 712/723 [01:53<00:00, 14.94it/s]
Loading 0:  99%|█████████▉| 715/723 [01:53<00:00, 15.87it/s]
Loading 0:  99%|█████████▉| 718/723 [01:53<00:00, 17.43it/s]
Loading 0: 100%|█████████▉| 721/723 [01:53<00:00, 19.43it/s]
Loading 0: 100%|█████████▉| 722/723 [02:04<00:00, 19.43it/s]
Loading 0: 100%|██████████| 723/723 [02:04<00:00,  1.20s/it]
                                                            
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v29-mkmlizer:   warnings.warn(
undi95-meta-llama-3-70b-6209-v29-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v29-mkmlizer:   warnings.warn(
undi95-meta-llama-3-70b-6209-v29-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v29-mkmlizer:   warnings.warn(
undi95-meta-llama-3-70b-6209-v29-mkmlizer: 
Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading shards:  50%|█████     | 1/2 [00:05<00:05,  5.97s/it]
Downloading shards: 100%|██████████| 2/2 [00:09<00:00,  4.28s/it]
Downloading shards: 100%|██████████| 2/2 [00:09<00:00,  4.53s/it]
undi95-meta-llama-3-70b-6209-v29-mkmlizer: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.12it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.56it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.23it/s]
undi95-meta-llama-3-70b-6209-v29-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
undi95-meta-llama-3-70b-6209-v29-mkmlizer: Saving duration: 1.414s
undi95-meta-llama-3-70b-6209-v29-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 14.293s
undi95-meta-llama-3-70b-6209-v29-mkmlizer: creating bucket guanaco-reward-models
undi95-meta-llama-3-70b-6209-v29-mkmlizer: Bucket 's3://guanaco-reward-models/' created
undi95-meta-llama-3-70b-6209-v29-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v29_reward
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v29_reward/merges.txt
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v29_reward/vocab.json
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v29_reward/tokenizer.json
undi95-meta-llama-3-70b-6209-v29-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v29_reward/reward.tensors
Job undi95-meta-llama-3-70b-6209-v29-mkmlizer completed after 866.7s with status: succeeded
Stopping job with name undi95-meta-llama-3-70b-6209-v29-mkmlizer
Pipeline stage MKMLizer completed in 867.93s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service undi95-meta-llama-3-70b-6209-v29
Waiting for inference service undi95-meta-llama-3-70b-6209-v29 to be ready
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Inference service undi95-meta-llama-3-70b-6209-v29 ready after 101.31579184532166s
Pipeline stage ISVCDeployer completed in 103.06s
Running pipeline stage StressChecker
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Received healthy response to inference request in 5.153790473937988s
Received healthy response to inference request in 4.283852815628052s
Received healthy response to inference request in 4.22096848487854s
Received healthy response to inference request in 4.3299880027771s
Received healthy response to inference request in 4.283044815063477s
5 requests
0 failed requests
5th percentile: 4.233383750915527
10th percentile: 4.245799016952515
20th percentile: 4.27062954902649
30th percentile: 4.283206415176392
40th percentile: 4.283529615402221
50th percentile: 4.283852815628052
60th percentile: 4.302306890487671
70th percentile: 4.32076096534729
80th percentile: 4.494748497009278
90th percentile: 4.824269485473633
95th percentile: 4.98902997970581
99th percentile: 5.1208383750915525
mean time: 4.454328918457032
Pipeline stage StressChecker completed in 23.17s
undi95-meta-llama-3-70b_6209_v29 status is now deployed due to DeploymentManager action
undi95-meta-llama-3-70b_6209_v29 status is now inactive due to admin request
admin requested tearing down of undi95-meta-llama-3-70b_6209_v29
Running pipeline stage ISVCDeleter
Checking if service undi95-meta-llama-3-70b-6209-v29 is running
Tearing down inference service undi95-meta-llama-3-70b-6209-v29
Service undi95-meta-llama-3-70b-6209-v29 has been torndown
Pipeline stage ISVCDeleter completed in 5.02s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key undi95-meta-llama-3-70b-6209-v29/config.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/flywheel_model.1.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/flywheel_model.2.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/flywheel_model.3.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/flywheel_model.4.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/flywheel_model.5.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/tokenizer.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v29/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key undi95-meta-llama-3-70b-6209-v29_reward/config.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v29_reward/merges.txt from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v29_reward/reward.tensors from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v29_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v29_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v29_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v29_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 12.66s
undi95-meta-llama-3-70b_6209_v29 status is now torndown due to DeploymentManager action