undi95-meta-llama-3-70b_6209

developer_uid: Jellywibble
submission_id: undi95-meta-llama-3-70b_6209_v30
model_name: meta-llama-1500k
model_group: Undi95/Meta-Llama-3-70B-
status: torndown
timestamp: 2024-07-25T17:49:13+00:00
num_battles: 177504
num_wins: 85959
celo_rating: 1212.65
family_friendly_score: 0.0
submission_type: basic
model_repo: Undi95/Meta-Llama-3-70B-Instruct-hf
model_architecture: LlamaForCausalLM
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
model_num_parameters: 70553706496.0
best_of: 2
max_input_tokens: 1500
max_output_tokens: 64
display_name: meta-llama-1500k
is_internal_developer: True
language_model: Undi95/Meta-Llama-3-70B-Instruct-hf
model_size: 71B
ranking_group: single
us_pacific_date: 2024-07-25
win_ratio: 0.4842651433207139
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 50, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|im_end|>', '<|im_start|>', '\n\n'], 'max_input_tokens': 1500, 'best_of': 2, 'max_output_tokens': 64, 'reward_max_token_input': 256}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s Persona: {memory}<|im_end|>\n", 'prompt_template': '<|im_start|>system\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "''", 'prompt_template': "''", 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
Resubmit model
Running pipeline stage MKMLizer
Starting job with name undi95-meta-llama-3-70b-6209-v30-mkmlizer
Waiting for job on undi95-meta-llama-3-70b-6209-v30-mkmlizer to finish
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║     _____            __           __                                ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║       /___/                                                         ║
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║                                                                     ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  Version: 0.9.7                                                     ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  https://mk1.ai                                                     ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║                                                                     ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  The license key for the current software has been verified as      ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  belonging to:                                                      ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║                                                                     ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  Chai Research Corp.                                                ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ║                                                                     ║
undi95-meta-llama-3-70b-6209-v30-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission blend_tefuf_2024-07-12: ('http://chaiml-sao10k-l3-rp-v3-3-v37-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:52664->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
undi95-meta-llama-3-70b-6209-v30-mkmlizer: Downloaded to shared memory in 275.559s
undi95-meta-llama-3-70b-6209-v30-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpu6cy90wi, device:0
undi95-meta-llama-3-70b-6209-v30-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
undi95-meta-llama-3-70b-6209-v30-mkmlizer: 
Loading 0:   0%|          | 0/723 [00:00<?, ?it/s]
Loading 0:   0%|          | 3/723 [00:00<01:35,  7.57it/s]
Loading 0:   1%|          | 4/723 [00:00<02:25,  4.94it/s]
Loading 0:   1%|          | 5/723 [00:01<03:16,  3.66it/s]
Loading 0:   1%|          | 8/723 [00:01<01:38,  7.27it/s]
Loading 0:   1%|▏         | 10/723 [00:01<01:25,  8.31it/s]
Loading 0:   2%|▏         | 12/723 [00:01<01:40,  7.11it/s]
Loading 0:   2%|▏         | 13/723 [00:01<01:34,  7.52it/s]
Loading 0:   2%|▏         | 14/723 [00:02<01:29,  7.94it/s]
Loading 0:   2%|▏         | 16/723 [00:02<01:20,  8.79it/s]
Loading 0:   2%|▏         | 18/723 [00:03<02:38,  4.44it/s]
Loading 0:   3%|▎         | 21/723 [00:03<02:09,  5.42it/s]
Loading 0:   3%|▎         | 22/723 [00:03<02:35,  4.52it/s]
Loading 0:   3%|▎         | 23/723 [00:04<03:06,  3.75it/s]
Loading 0:   4%|▎         | 26/723 [00:04<01:55,  6.06it/s]
Loading 0:   4%|▍         | 28/723 [00:04<01:39,  6.95it/s]
Loading 0:   4%|▍         | 30/723 [00:05<01:50,  6.28it/s]
Loading 0:   4%|▍         | 31/723 [00:05<02:17,  5.04it/s]
Loading 0:   4%|▍         | 32/723 [00:05<02:54,  3.95it/s]
Loading 0:   5%|▍         | 35/723 [00:06<01:46,  6.43it/s]
Loading 0:   5%|▌         | 37/723 [00:06<01:34,  7.29it/s]
Loading 0:   5%|▌         | 39/723 [00:06<01:18,  8.71it/s]
Loading 0:   6%|▌         | 41/723 [00:06<01:14,  9.15it/s]
Loading 0:   6%|▌         | 43/723 [00:07<01:44,  6.51it/s]
Loading 0:   6%|▌         | 44/723 [00:07<02:14,  5.05it/s]
Loading 0:   6%|▌         | 45/723 [00:07<02:49,  4.00it/s]
Loading 0:   7%|▋         | 48/723 [00:08<02:12,  5.10it/s]
Loading 0:   7%|▋         | 49/723 [00:08<02:36,  4.29it/s]
Loading 0:   7%|▋         | 50/723 [00:09<03:09,  3.55it/s]
Loading 0:   7%|▋         | 53/723 [00:09<01:53,  5.88it/s]
Loading 0:   7%|▋         | 54/723 [00:09<01:45,  6.35it/s]
Loading 0:   8%|▊         | 57/723 [00:09<01:40,  6.64it/s]
Loading 0:   8%|▊         | 58/723 [00:10<02:06,  5.25it/s]
Loading 0:   8%|▊         | 59/723 [00:10<02:42,  4.09it/s]
Loading 0:   9%|▊         | 62/723 [00:10<01:40,  6.55it/s]
Loading 0:   9%|▉         | 64/723 [00:10<01:29,  7.36it/s]
Loading 0:   9%|▉         | 66/723 [00:11<01:14,  8.77it/s]
Loading 0:   9%|▉         | 68/723 [00:11<01:11,  9.21it/s]
Loading 0:  10%|▉         | 70/723 [00:12<02:07,  5.12it/s]
Loading 0:  10%|▉         | 71/723 [00:12<02:37,  4.13it/s]
Loading 0:  10%|█         | 73/723 [00:12<01:56,  5.56it/s]
Loading 0:  10%|█         | 75/723 [00:13<02:00,  5.38it/s]
Loading 0:  11%|█         | 76/723 [00:13<02:25,  4.44it/s]
Loading 0:  11%|█         | 77/723 [00:13<02:56,  3.66it/s]
Loading 0:  11%|█         | 80/723 [00:14<01:44,  6.15it/s]
Loading 0:  11%|█▏        | 82/723 [00:14<01:30,  7.12it/s]
Loading 0:  12%|█▏        | 84/723 [00:14<01:39,  6.40it/s]
Loading 0:  12%|█▏        | 85/723 [00:15<02:06,  5.03it/s]
Loading 0:  12%|█▏        | 86/723 [00:15<02:39,  3.98it/s]
Loading 0:  12%|█▏        | 89/723 [00:15<01:37,  6.52it/s]
Loading 0:  13%|█▎        | 91/723 [00:15<01:25,  7.41it/s]
Loading 0:  13%|█▎        | 93/723 [00:16<01:47,  5.84it/s]
Loading 0:  13%|█▎        | 94/723 [00:16<02:13,  4.72it/s]
Loading 0:  13%|█▎        | 95/723 [00:17<02:44,  3.82it/s]
Loading 0:  14%|█▎        | 98/723 [00:17<01:40,  6.24it/s]
Loading 0:  14%|█▍        | 100/723 [00:17<01:27,  7.14it/s]
Loading 0:  14%|█▍        | 102/723 [00:17<01:38,  6.32it/s]
Loading 0:  14%|█▍        | 103/723 [00:18<02:04,  4.97it/s]
Loading 0:  14%|█▍        | 104/723 [00:18<02:35,  3.99it/s]
Loading 0:  15%|█▍        | 107/723 [00:18<01:34,  6.51it/s]
Loading 0:  15%|█▌        | 109/723 [00:19<01:23,  7.39it/s]
Loading 0:  15%|█▌        | 111/723 [00:19<02:18,  4.41it/s]
Loading 0:  16%|█▌        | 113/723 [00:20<01:48,  5.63it/s]
Loading 0:  16%|█▌        | 115/723 [00:20<01:34,  6.46it/s]
Loading 0:  16%|█▌        | 117/723 [00:20<01:51,  5.43it/s]
Loading 0:  17%|█▋        | 120/723 [00:21<01:38,  6.10it/s]
Loading 0:  17%|█▋        | 121/723 [00:21<02:00,  5.01it/s]
Loading 0:  17%|█▋        | 122/723 [00:21<02:27,  4.06it/s]
Loading 0:  17%|█▋        | 125/723 [00:39<26:37,  2.67s/it]
Loading 0:  17%|█▋        | 126/723 [00:39<22:10,  2.23s/it]
Loading 0:  18%|█▊        | 127/723 [00:39<17:57,  1.81s/it]
Loading 0:  18%|█▊        | 129/723 [00:39<12:08,  1.23s/it]
Loading 0:  18%|█▊        | 130/723 [00:40<10:23,  1.05s/it]
Loading 0:  18%|█▊        | 131/723 [00:40<09:02,  1.09it/s]
Loading 0:  19%|█▊        | 134/723 [00:40<04:46,  2.06it/s]
Loading 0:  19%|█▉        | 136/723 [00:41<03:33,  2.75it/s]
Loading 0:  19%|█▉        | 137/723 [00:41<03:36,  2.70it/s]
Loading 0:  19%|█▉        | 139/723 [00:41<02:34,  3.78it/s]
Loading 0:  19%|█▉        | 140/723 [00:41<02:16,  4.26it/s]
Loading 0:  20%|█▉        | 142/723 [00:41<01:47,  5.40it/s]
Loading 0:  20%|█▉        | 143/723 [00:42<02:12,  4.37it/s]
Loading 0:  20%|█▉        | 144/723 [00:42<02:43,  3.54it/s]
Loading 0:  20%|██        | 147/723 [00:43<02:00,  4.79it/s]
Loading 0:  20%|██        | 148/723 [00:43<02:20,  4.09it/s]
Loading 0:  21%|██        | 149/723 [00:43<02:48,  3.40it/s]
Loading 0:  21%|██        | 152/723 [00:44<01:40,  5.69it/s]
Loading 0:  21%|██        | 153/723 [00:44<01:34,  6.06it/s]
Loading 0:  22%|██▏       | 156/723 [00:44<01:32,  6.10it/s]
Loading 0:  22%|██▏       | 157/723 [00:45<01:53,  4.99it/s]
Loading 0:  22%|██▏       | 158/723 [00:45<02:23,  3.95it/s]
Loading 0:  22%|██▏       | 161/723 [00:45<01:28,  6.35it/s]
Loading 0:  23%|██▎       | 163/723 [00:45<01:19,  7.04it/s]
Loading 0:  23%|██▎       | 165/723 [00:46<01:06,  8.38it/s]
Loading 0:  23%|██▎       | 167/723 [00:46<01:02,  8.85it/s]
Loading 0:  23%|██▎       | 169/723 [00:46<01:29,  6.16it/s]
Loading 0:  24%|██▎       | 170/723 [00:47<01:55,  4.80it/s]
Loading 0:  24%|██▎       | 171/723 [00:47<02:25,  3.78it/s]
Loading 0:  24%|██▍       | 174/723 [00:48<01:53,  4.84it/s]
Loading 0:  24%|██▍       | 175/723 [00:48<02:14,  4.08it/s]
Loading 0:  24%|██▍       | 176/723 [00:49<02:40,  3.41it/s]
Loading 0:  25%|██▍       | 179/723 [00:49<01:36,  5.67it/s]
Loading 0:  25%|██▍       | 180/723 [00:49<01:30,  6.02it/s]
Loading 0:  25%|██▌       | 183/723 [00:49<01:26,  6.23it/s]
Loading 0:  25%|██▌       | 184/723 [00:50<01:46,  5.07it/s]
Loading 0:  26%|██▌       | 185/723 [00:50<02:12,  4.06it/s]
Loading 0:  26%|██▌       | 188/723 [00:50<01:22,  6.47it/s]
Loading 0:  26%|██▋       | 190/723 [00:50<01:13,  7.25it/s]
Loading 0:  27%|██▋       | 192/723 [00:51<01:01,  8.57it/s]
Loading 0:  27%|██▋       | 194/723 [00:51<01:01,  8.66it/s]
Loading 0:  27%|██▋       | 196/723 [00:52<01:49,  4.80it/s]
Loading 0:  27%|██▋       | 197/723 [00:52<02:15,  3.87it/s]
Loading 0:  28%|██▊       | 199/723 [00:52<01:40,  5.20it/s]
Loading 0:  28%|██▊       | 201/723 [00:53<01:43,  5.06it/s]
Loading 0:  28%|██▊       | 202/723 [00:53<02:04,  4.19it/s]
Loading 0:  28%|██▊       | 203/723 [00:54<02:27,  3.52it/s]
Loading 0:  28%|██▊       | 206/723 [00:54<01:27,  5.88it/s]
Loading 0:  29%|██▉       | 208/723 [00:54<01:16,  6.71it/s]
Loading 0:  29%|██▉       | 210/723 [00:54<01:25,  5.97it/s]
Loading 0:  29%|██▉       | 211/723 [00:55<01:48,  4.71it/s]
Loading 0:  29%|██▉       | 212/723 [00:55<02:16,  3.74it/s]
Loading 0:  30%|██▉       | 215/723 [00:55<01:23,  6.11it/s]
Loading 0:  30%|███       | 217/723 [00:55<01:12,  6.99it/s]
Loading 0:  30%|███       | 219/723 [00:56<01:31,  5.53it/s]
Loading 0:  30%|███       | 220/723 [00:56<01:51,  4.51it/s]
Loading 0:  31%|███       | 221/723 [00:57<02:17,  3.64it/s]
Loading 0:  31%|███       | 224/723 [00:57<01:23,  5.97it/s]
Loading 0:  31%|███▏      | 226/723 [00:57<01:12,  6.84it/s]
Loading 0:  32%|███▏      | 228/723 [00:58<01:21,  6.09it/s]
Loading 0:  32%|███▏      | 229/723 [00:58<01:42,  4.84it/s]
Loading 0:  32%|███▏      | 230/723 [00:59<02:08,  3.83it/s]
Loading 0:  32%|███▏      | 233/723 [00:59<01:18,  6.25it/s]
Loading 0:  33%|███▎      | 235/723 [00:59<01:08,  7.10it/s]
Loading 0:  33%|███▎      | 237/723 [01:00<01:53,  4.29it/s]
Loading 0:  33%|███▎      | 239/723 [01:00<01:28,  5.48it/s]
Loading 0:  33%|███▎      | 241/723 [01:00<01:15,  6.36it/s]
Loading 0:  34%|███▎      | 243/723 [01:01<01:29,  5.36it/s]
Loading 0:  34%|███▍      | 246/723 [01:01<01:17,  6.12it/s]
Loading 0:  34%|███▍      | 247/723 [01:01<01:36,  4.93it/s]
Loading 0:  34%|███▍      | 248/723 [01:02<02:01,  3.91it/s]
Loading 0:  35%|███▍      | 251/723 [01:02<01:17,  6.12it/s]
Loading 0:  35%|███▍      | 253/723 [01:02<01:07,  6.95it/s]
Loading 0:  35%|███▌      | 255/723 [01:03<01:16,  6.15it/s]
Loading 0:  35%|███▌      | 256/723 [01:03<01:35,  4.90it/s]
Loading 0:  36%|███▌      | 257/723 [01:03<01:56,  3.99it/s]
Loading 0:  36%|███▌      | 260/723 [01:04<01:11,  6.49it/s]
Loading 0:  36%|███▌      | 262/723 [01:04<01:02,  7.40it/s]
Loading 0:  37%|███▋      | 264/723 [01:04<01:10,  6.51it/s]
Loading 0:  37%|███▋      | 265/723 [01:04<01:06,  6.88it/s]
Loading 0:  37%|███▋      | 266/723 [01:04<01:02,  7.26it/s]
Loading 0:  37%|███▋      | 268/723 [01:05<00:56,  8.12it/s]
Loading 0:  37%|███▋      | 269/723 [01:24<31:59,  4.23s/it]
Loading 0:  37%|███▋      | 270/723 [01:25<25:26,  3.37s/it]
Loading 0:  38%|███▊      | 273/723 [01:25<13:16,  1.77s/it]
Loading 0:  38%|███▊      | 274/723 [01:26<11:16,  1.51s/it]
Loading 0:  38%|███▊      | 275/723 [01:26<09:33,  1.28s/it]
Loading 0:  38%|███▊      | 278/723 [01:26<05:06,  1.45it/s]
Loading 0:  39%|███▊      | 280/723 [01:27<03:42,  1.99it/s]
Loading 0:  39%|███▉      | 282/723 [01:27<03:00,  2.44it/s]
Loading 0:  39%|███▉      | 283/723 [01:27<02:58,  2.47it/s]
Loading 0:  39%|███▉      | 284/723 [01:28<03:03,  2.39it/s]
Loading 0:  40%|███▉      | 287/723 [01:28<01:46,  4.10it/s]
Loading 0:  40%|███▉      | 289/723 [01:28<01:26,  5.00it/s]
Loading 0:  40%|████      | 291/723 [01:28<01:09,  6.26it/s]
Loading 0:  41%|████      | 293/723 [01:28<01:00,  7.13it/s]
Loading 0:  41%|████      | 295/723 [01:29<01:16,  5.61it/s]
Loading 0:  41%|████      | 296/723 [01:29<01:33,  4.56it/s]
Loading 0:  41%|████      | 297/723 [01:30<01:55,  3.69it/s]
Loading 0:  41%|████▏     | 300/723 [01:30<01:28,  4.79it/s]
Loading 0:  42%|████▏     | 301/723 [01:31<01:42,  4.11it/s]
Loading 0:  42%|████▏     | 302/723 [01:31<02:00,  3.49it/s]
Loading 0:  42%|████▏     | 305/723 [01:31<01:12,  5.75it/s]
Loading 0:  42%|████▏     | 306/723 [01:31<01:08,  6.13it/s]
Loading 0:  43%|████▎     | 309/723 [01:32<01:07,  6.17it/s]
Loading 0:  43%|████▎     | 310/723 [01:32<01:24,  4.87it/s]
Loading 0:  43%|████▎     | 311/723 [01:33<01:46,  3.87it/s]
Loading 0:  43%|████▎     | 314/723 [01:33<01:05,  6.24it/s]
Loading 0:  44%|████▎     | 316/723 [01:33<00:57,  7.11it/s]
Loading 0:  44%|████▍     | 318/723 [01:33<00:48,  8.43it/s]
Loading 0:  44%|████▍     | 320/723 [01:33<00:44,  9.00it/s]
Loading 0:  45%|████▍     | 322/723 [01:34<01:21,  4.93it/s]
Loading 0:  45%|████▍     | 323/723 [01:35<01:40,  3.98it/s]
Loading 0:  45%|████▍     | 325/723 [01:35<01:14,  5.37it/s]
Loading 0:  45%|████▌     | 327/723 [01:35<01:15,  5.27it/s]
Loading 0:  45%|████▌     | 328/723 [01:36<01:28,  4.48it/s]
Loading 0:  46%|████▌     | 329/723 [01:36<01:45,  3.74it/s]
Loading 0:  46%|████▌     | 332/723 [01:36<01:02,  6.29it/s]
Loading 0:  46%|████▌     | 334/723 [01:36<00:52,  7.39it/s]
Loading 0:  46%|████▋     | 336/723 [01:37<00:58,  6.56it/s]
Loading 0:  47%|████▋     | 337/723 [01:37<01:16,  5.01it/s]
Loading 0:  47%|████▋     | 338/723 [01:38<01:37,  3.93it/s]
Loading 0:  47%|████▋     | 341/723 [01:38<00:59,  6.40it/s]
Loading 0:  47%|████▋     | 343/723 [01:38<00:52,  7.25it/s]
Loading 0:  48%|████▊     | 345/723 [01:38<01:06,  5.68it/s]
Loading 0:  48%|████▊     | 346/723 [01:39<01:22,  4.57it/s]
Loading 0:  48%|████▊     | 347/723 [01:39<01:41,  3.70it/s]
Loading 0:  48%|████▊     | 350/723 [01:39<01:01,  6.08it/s]
Loading 0:  49%|████▊     | 352/723 [01:40<00:53,  6.97it/s]
Loading 0:  49%|████▉     | 354/723 [01:40<01:00,  6.15it/s]
Loading 0:  49%|████▉     | 355/723 [01:40<01:16,  4.80it/s]
Loading 0:  49%|████▉     | 356/723 [01:41<01:36,  3.82it/s]
Loading 0:  50%|████▉     | 359/723 [01:41<00:58,  6.24it/s]
Loading 0:  50%|████▉     | 361/723 [01:41<00:50,  7.10it/s]
Loading 0:  50%|█████     | 363/723 [01:42<01:25,  4.22it/s]
Loading 0:  50%|█████     | 365/723 [01:42<01:06,  5.37it/s]
Loading 0:  51%|█████     | 366/723 [01:42<01:02,  5.74it/s]
Loading 0:  51%|█████     | 368/723 [01:43<00:52,  6.76it/s]
Loading 0:  51%|█████     | 369/723 [01:43<01:10,  5.03it/s]
Loading 0:  51%|█████▏    | 372/723 [01:43<01:00,  5.84it/s]
Loading 0:  52%|█████▏    | 373/723 [01:44<01:14,  4.68it/s]
Loading 0:  52%|█████▏    | 374/723 [01:44<01:33,  3.74it/s]
Loading 0:  52%|█████▏    | 377/723 [01:44<00:56,  6.11it/s]
Loading 0:  52%|█████▏    | 379/723 [01:45<00:49,  6.99it/s]
Loading 0:  53%|█████▎    | 381/723 [01:45<00:55,  6.14it/s]
Loading 0:  53%|█████▎    | 382/723 [01:45<01:10,  4.82it/s]
Loading 0:  53%|█████▎    | 383/723 [01:46<01:26,  3.95it/s]
Loading 0:  53%|█████▎    | 386/723 [01:46<00:52,  6.46it/s]
Loading 0:  54%|█████▎    | 388/723 [01:46<00:45,  7.29it/s]
Loading 0:  54%|█████▍    | 390/723 [01:47<00:53,  6.22it/s]
Loading 0:  54%|█████▍    | 391/723 [01:47<00:50,  6.52it/s]
Loading 0:  54%|█████▍    | 392/723 [01:47<00:48,  6.83it/s]
Loading 0:  54%|█████▍    | 394/723 [01:47<00:41,  7.85it/s]
Loading 0:  55%|█████▍    | 395/723 [01:47<01:01,  5.35it/s]
Loading 0:  55%|█████▍    | 396/723 [01:48<01:23,  3.93it/s]
Loading 0:  55%|█████▌    | 399/723 [01:48<01:03,  5.08it/s]
Loading 0:  55%|█████▌    | 400/723 [01:49<01:16,  4.20it/s]
Loading 0:  55%|█████▌    | 401/723 [02:09<24:18,  4.53s/it]
Loading 0:  56%|█████▌    | 404/723 [02:09<12:40,  2.38s/it]
Loading 0:  56%|█████▌    | 405/723 [02:09<10:25,  1.97s/it]
Loading 0:  56%|█████▋    | 408/723 [02:10<06:08,  1.17s/it]
Loading 0:  57%|█████▋    | 409/723 [02:10<05:26,  1.04s/it]
Loading 0:  57%|█████▋    | 410/723 [02:11<04:51,  1.07it/s]
Loading 0:  57%|█████▋    | 413/723 [02:11<02:42,  1.91it/s]
Loading 0:  57%|█████▋    | 415/723 [02:11<02:01,  2.53it/s]
Loading 0:  58%|█████▊    | 417/723 [02:11<01:30,  3.37it/s]
Loading 0:  58%|█████▊    | 419/723 [02:11<01:12,  4.17it/s]
Loading 0:  58%|█████▊    | 421/723 [02:12<01:15,  4.00it/s]
Loading 0:  58%|█████▊    | 422/723 [02:12<01:24,  3.56it/s]
Loading 0:  59%|█████▊    | 423/723 [02:13<01:37,  3.08it/s]
Loading 0:  59%|█████▉    | 426/723 [02:13<01:10,  4.23it/s]
Loading 0:  59%|█████▉    | 427/723 [02:14<01:19,  3.74it/s]
Loading 0:  59%|█████▉    | 428/723 [02:14<01:32,  3.19it/s]
Loading 0:  60%|█████▉    | 431/723 [02:14<00:55,  5.31it/s]
Loading 0:  60%|█████▉    | 432/723 [02:15<00:5
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
undi95-meta-llama-3-70b-6209-v30-mkmlizer: quantized model in 227.689s
undi95-meta-llama-3-70b-6209-v30-mkmlizer: Processed model Undi95/Meta-Llama-3-70B-Instruct-hf in 503.249s
undi95-meta-llama-3-70b-6209-v30-mkmlizer: creating bucket guanaco-mkml-models
undi95-meta-llama-3-70b-6209-v30-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
undi95-meta-llama-3-70b-6209-v30-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/config.json
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/special_tokens_map.json
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/tokenizer_config.json
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/tokenizer.json
Failed to get response for submission mistralai-mixtral-8x7b-_3473_v86: ('http://mistralai-mixtral-8x7b-3473-v86-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:54588->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/flywheel_model.5.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/flywheel_model.5.safetensors
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/flywheel_model.0.safetensors
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/flywheel_model.2.safetensors
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/flywheel_model.4.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/flywheel_model.4.safetensors
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/flywheel_model.1.safetensors
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v30/flywheel_model.3.safetensors
undi95-meta-llama-3-70b-6209-v30-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
undi95-meta-llama-3-70b-6209-v30-mkmlizer: 1,  5.67it/s]
Loading 0:  60%|██████    | 435/723 [02:15<00:49,  5.81it/s]
Loading 0:  60%|██████    | 436/723 [02:15<01:01,  4.63it/s]
Loading 0:  60%|██████    | 437/723 [02:16<01:17,  3.69it/s]
Loading 0:  61%|██████    | 440/723 [02:16<00:47,  5.96it/s]
Loading 0:  61%|██████    | 442/723 [02:16<00:41,  6.74it/s]
Loading 0:  61%|██████▏   | 444/723 [02:16<00:34,  8.00it/s]
Loading 0:  62%|██████▏   | 446/723 [02:17<00:32,  8.55it/s]
Loading 0:  62%|██████▏   | 448/723 [02:17<00:56,  4.88it/s]
Loading 0:  62%|██████▏   | 449/723 [02:18<01:09,  3.95it/s]
Loading 0:  62%|██████▏   | 451/723 [02:18<00:51,  5.33it/s]
Loading 0:  63%|██████▎   | 453/723 [02:18<00:51,  5.20it/s]
Loading 0:  63%|██████▎   | 454/723 [02:19<01:02,  4.31it/s]
Loading 0:  63%|██████▎   | 455/723 [02:19<01:15,  3.55it/s]
Loading 0:  63%|██████▎   | 458/723 [02:19<00:44,  5.90it/s]
Loading 0:  64%|██████▎   | 460/723 [02:20<00:39,  6.73it/s]
Loading 0:  64%|██████▍   | 462/723 [02:20<00:42,  6.17it/s]
Loading 0:  64%|██████▍   | 463/723 [02:20<00:52,  4.97it/s]
Loading 0:  64%|██████▍   | 464/723 [02:21<00:55,  4.66it/s]
Loading 0:  65%|██████▍   | 469/723 [02:21<00:25,  9.78it/s]
Loading 0:  65%|██████▌   | 471/723 [02:21<00:25,  9.78it/s]
Loading 0:  65%|██████▌   | 473/723 [02:21<00:25,  9.84it/s]
Loading 0:  66%|██████▌   | 478/723 [02:21<00:15, 15.54it/s]
Loading 0:  67%|██████▋   | 481/723 [02:21<00:14, 16.35it/s]
Loading 0:  67%|██████▋   | 484/723 [02:22<00:13, 17.57it/s]
Loading 0:  67%|██████▋   | 487/723 [02:22<00:12, 19.51it/s]
Loading 0:  68%|██████▊   | 490/723 [02:22<00:13, 17.06it/s]
Loading 0:  68%|██████▊   | 493/723 [02:22<00:12, 19.16it/s]
Loading 0:  69%|██████▊   | 496/723 [02:22<00:13, 16.67it/s]
Loading 0:  69%|██████▉   | 499/723 [02:22<00:12, 17.98it/s]
Loading 0:  69%|██████▉   | 501/723 [02:23<00:12, 17.27it/s]
Loading 0:  70%|██████▉   | 505/723 [02:23<00:10, 21.12it/s]
Loading 0:  70%|███████   | 508/723 [02:23<00:10, 20.28it/s]
Loading 0:  71%|███████   | 511/723 [02:23<00:10, 20.56it/s]
Loading 0:  71%|███████   | 514/723 [02:23<00:09, 21.88it/s]
Loading 0:  72%|███████▏  | 517/723 [02:23<00:08, 23.18it/s]
Loading 0:  72%|███████▏  | 520/723 [02:24<00:11, 17.54it/s]
Loading 0:  72%|███████▏  | 523/723 [02:24<00:12, 16.55it/s]
Loading 0:  73%|███████▎  | 526/723 [02:24<00:11, 17.35it/s]
Loading 0:  73%|███████▎  | 528/723 [02:24<00:11, 16.74it/s]
Loading 0:  74%|███████▎  | 532/723 [02:24<00:09, 20.29it/s]
Loading 0:  74%|███████▍  | 535/723 [02:24<00:09, 19.60it/s]
Loading 0:  74%|███████▍  | 535/723 [02:40<00:09, 19.60it/s]
Loading 0:  74%|███████▍  | 536/723 [02:45<07:58,  2.56s/it]
Loading 0:  75%|███████▍  | 540/723 [02:45<04:38,  1.52s/it]
Loading 0:  75%|███████▌  | 543/723 [02:45<03:13,  1.07s/it]
Loading 0:  76%|███████▌  | 546/723 [02:45<02:16,  1.29it/s]
Loading 0:  76%|███████▌  | 548/723 [02:46<01:47,  1.63it/s]
Loading 0:  76%|███████▌  | 550/723 [02:46<01:23,  2.08it/s]
Loading 0:  76%|███████▋  | 553/723 [02:46<00:56,  2.99it/s]
Loading 0:  77%|███████▋  | 555/723 [02:46<00:45,  3.72it/s]
Loading 0:  77%|███████▋  | 559/723 [02:46<00:28,  5.82it/s]
Loading 0:  78%|███████▊  | 562/723 [02:46<00:21,  7.38it/s]
Loading 0:  78%|███████▊  | 564/723 [02:46<00:19,  8.14it/s]
Loading 0:  79%|███████▊  | 568/723 [02:47<00:13, 11.43it/s]
Loading 0:  79%|███████▉  | 571/723 [02:47<00:10, 13.84it/s]
Loading 0:  79%|███████▉  | 574/723 [02:47<00:12, 12.23it/s]
Loading 0:  80%|███████▉  | 576/723 [02:47<00:11, 12.72it/s]
Loading 0:  80%|████████  | 579/723 [02:47<00:09, 15.46it/s]
Loading 0:  80%|████████  | 582/723 [02:47<00:09, 14.87it/s]
Loading 0:  81%|████████  | 586/723 [02:48<00:07, 18.33it/s]
Loading 0:  81%|████████▏ | 589/723 [02:48<00:07, 18.50it/s]
Loading 0:  82%|████████▏ | 592/723 [02:48<00:06, 18.80it/s]
Loading 0:  82%|████████▏ | 595/723 [02:48<00:06, 19.91it/s]
Loading 0:  83%|████████▎ | 598/723 [02:48<00:08, 14.67it/s]
Loading 0:  83%|████████▎ | 600/723 [02:48<00:08, 14.62it/s]
Loading 0:  84%|████████▎ | 604/723 [02:49<00:06, 18.42it/s]
Loading 0:  84%|████████▍ | 607/723 [02:49<00:06, 18.73it/s]
Loading 0:  84%|████████▍ | 610/723 [02:49<00:05, 19.24it/s]
Loading 0:  85%|████████▍ | 613/723 [02:49<00:05, 20.80it/s]
Loading 0:  85%|████████▌ | 616/723 [02:49<00:06, 17.57it/s]
Loading 0:  86%|████████▌ | 619/723 [02:49<00:05, 18.89it/s]
Loading 0:  86%|████████▌ | 622/723 [02:50<00:06, 16.32it/s]
Loading 0:  86%|████████▋ | 625/723 [02:50<00:05, 17.74it/s]
Loading 0:  87%|████████▋ | 627/723 [02:50<00:05, 16.98it/s]
Loading 0:  87%|████████▋ | 631/723 [02:50<00:04, 20.57it/s]
Loading 0:  88%|████████▊ | 634/723 [02:50<00:04, 20.28it/s]
Loading 0:  88%|████████▊ | 637/723 [02:50<00:04, 20.46it/s]
Loading 0:  89%|████████▊ | 640/723 [02:50<00:03, 21.67it/s]
Loading 0:  89%|████████▉ | 643/723 [02:51<00:03, 22.99it/s]
Loading 0:  89%|████████▉ | 646/723 [02:51<00:04, 17.35it/s]
Loading 0:  90%|████████▉ | 648/723 [02:51<00:05, 14.83it/s]
Loading 0:  90%|█████████ | 652/723 [02:51<00:04, 17.71it/s]
Loading 0:  90%|█████████ | 654/723 [02:51<00:04, 16.93it/s]
Loading 0:  91%|█████████ | 658/723 [02:51<00:03, 20.26it/s]
Loading 0:  91%|█████████▏| 661/723 [02:52<00:03, 19.77it/s]
Loading 0:  92%|█████████▏| 664/723 [02:52<00:02, 20.17it/s]
Loading 0:  92%|█████████▏| 667/723 [02:52<00:02, 21.69it/s]
Loading 0:  93%|█████████▎| 671/723 [02:52<00:02, 24.61it/s]
Loading 0:  93%|█████████▎| 674/723 [02:52<00:02, 17.56it/s]
Loading 0:  93%|█████████▎| 674/723 [03:13<00:02, 17.56it/s]
Loading 0:  93%|█████████▎| 675/723 [03:13<01:58,  2.47s/it]
Loading 0:  94%|█████████▍| 678/723 [03:13<01:15,  1.67s/it]
Loading 0:  94%|█████████▍| 680/723 [03:13<00:55,  1.30s/it]
Loading 0:  95%|█████████▍| 685/723 [03:14<00:26,  1.41it/s]
Loading 0:  95%|█████████▌| 688/723 [03:14<00:18,  1.92it/s]
Loading 0:  96%|█████████▌| 691/723 [03:14<00:12,  2.60it/s]
Loading 0:  96%|█████████▌| 694/723 [03:14<00:08,  3.53it/s]
Loading 0:  97%|█████████▋| 698/723 [03:14<00:05,  4.85it/s]
Loading 0:  97%|█████████▋| 701/723 [03:15<00:03,  5.76it/s]
Loading 0:  98%|█████████▊| 705/723 [03:15<00:02,  8.13it/s]
Loading 0:  98%|█████████▊| 708/723 [03:15<00:01,  9.17it/s]
Loading 0:  98%|█████████▊| 712/723 [03:15<00:00, 11.92it/s]
Loading 0:  99%|█████████▉| 715/723 [03:15<00:00, 13.21it/s]
Loading 0:  99%|█████████▉| 718/723 [03:15<00:00, 14.83it/s]
Loading 0: 100%|█████████▉| 721/723 [03:15<00:00, 16.96it/s]
Loading 0: 100%|█████████▉| 722/723 [03:27<00:00, 16.96it/s]
Loading 0: 100%|██████████| 723/723 [03:27<00:00,  1.23s/it]
                                                            
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v30-mkmlizer:   warnings.warn(
undi95-meta-llama-3-70b-6209-v30-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v30-mkmlizer:   warnings.warn(
undi95-meta-llama-3-70b-6209-v30-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v30-mkmlizer:   warnings.warn(
undi95-meta-llama-3-70b-6209-v30-mkmlizer: 
Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading shards:  50%|█████     | 1/2 [00:05<00:05,  5.89s/it]
Downloading shards: 100%|██████████| 2/2 [00:09<00:00,  4.26s/it]
Downloading shards: 100%|██████████| 2/2 [00:09<00:00,  4.50s/it]
undi95-meta-llama-3-70b-6209-v30-mkmlizer: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.25it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.71it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.38it/s]
undi95-meta-llama-3-70b-6209-v30-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
undi95-meta-llama-3-70b-6209-v30-mkmlizer: Saving duration: 1.410s
undi95-meta-llama-3-70b-6209-v30-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 14.148s
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
undi95-meta-llama-3-70b-6209-v30-mkmlizer: Bucket 's3://guanaco-reward-models/' created
undi95-meta-llama-3-70b-6209-v30-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v30_reward
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v30_reward/config.json
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v30_reward/special_tokens_map.json
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v30_reward/tokenizer_config.json
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v30_reward/merges.txt
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v30_reward/vocab.json
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v30_reward/tokenizer.json
undi95-meta-llama-3-70b-6209-v30-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v30_reward/reward.tensors
Job undi95-meta-llama-3-70b-6209-v30-mkmlizer completed after 574.98s with status: succeeded
Stopping job with name undi95-meta-llama-3-70b-6209-v30-mkmlizer
Pipeline stage MKMLizer completed in 576.45s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.12s
Running pipeline stage ISVCDeployer
Creating inference service undi95-meta-llama-3-70b-6209-v30
Waiting for inference service undi95-meta-llama-3-70b-6209-v30 to be ready
Failed to get response for submission blend_banet_2024-07-25: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Inference service undi95-meta-llama-3-70b-6209-v30 ready after 90.738694190979s
Pipeline stage ISVCDeployer completed in 92.43s
Running pipeline stage StressChecker
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Received healthy response to inference request in 5.308619260787964s
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v29: ('http://undi95-meta-llama-3-70b-6209-v29-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Received healthy response to inference request in 4.296011209487915s
Failed to get response for submission undi95-meta-llama-3-70b_6209_v27: ('http://undi95-meta-llama-3-70b-6209-v27-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Received healthy response to inference request in 4.270711660385132s
Received healthy response to inference request in 4.288548707962036s
Received healthy response to inference request in 4.306547164916992s
5 requests
0 failed requests
5th percentile: 4.274279069900513
10th percentile: 4.277846479415894
20th percentile: 4.284981298446655
30th percentile: 4.290041208267212
40th percentile: 4.2930262088775635
50th percentile: 4.296011209487915
60th percentile: 4.3002255916595455
70th percentile: 4.304439973831177
80th percentile: 4.5069615840911865
90th percentile: 4.907790422439575
95th percentile: 5.1082048416137695
99th percentile: 5.268536376953125
mean time: 4.494087600708008
Pipeline stage StressChecker completed in 23.50s
undi95-meta-llama-3-70b_6209_v30 status is now deployed due to DeploymentManager action
undi95-meta-llama-3-70b_6209_v30 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of undi95-meta-llama-3-70b_6209_v30
Running pipeline stage ISVCDeleter
Checking if service undi95-meta-llama-3-70b-6209-v30 is running
Tearing down inference service undi95-meta-llama-3-70b-6209-v30
Service undi95-meta-llama-3-70b-6209-v30 has been torndown
Pipeline stage ISVCDeleter completed in 5.01s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key undi95-meta-llama-3-70b-6209-v30/config.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/flywheel_model.1.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/flywheel_model.2.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/flywheel_model.3.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/flywheel_model.4.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/flywheel_model.5.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/tokenizer.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v30/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key undi95-meta-llama-3-70b-6209-v30_reward/config.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v30_reward/merges.txt from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v30_reward/reward.tensors from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v30_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v30_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v30_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v30_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 13.17s
undi95-meta-llama-3-70b_6209_v30 status is now torndown due to DeploymentManager action