submission_id: undi95-meta-llama-3-70b_6209_v21
developer_uid: chai_backend_admin
alignment_samples: 0
best_of: 2
celo_rating: 1204.73
display_name: undi95-meta-llama-3-hollow
formatter: {'memory_template': '""', 'prompt_template': '""', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 50, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|im_end|>', '<|im_start|>', '\n\n'], 'max_input_tokens': 512, 'best_of': 2, 'max_output_tokens': 64}
is_internal_developer: True
language_model: Undi95/Meta-Llama-3-70B-Instruct-hf
max_input_tokens: 512
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: Undi95/Meta-Llama-3-70B-
model_name: undi95-meta-llama-3-hollow
model_num_parameters: 70553706496.0
model_repo: Undi95/Meta-Llama-3-70B-Instruct-hf
model_size: 71B
num_battles: 11278
num_wins: 5632
propriety_score: 0.719892952720785
propriety_total_count: 1121.0
ranking_group: single
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "''", 'prompt_template': "''", 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
status: torndown
submission_type: basic
timestamp: 2024-07-23T03:45:19+00:00
us_pacific_date: 2024-07-22
win_ratio: 0.4993793225749246
Download Preference Data
Resubmit model
Running pipeline stage MKMLizer
Starting job with name undi95-meta-llama-3-70b-6209-v21-mkmlizer
Waiting for job on undi95-meta-llama-3-70b-6209-v21-mkmlizer to finish
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ _____ __ __ ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ /___/ ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ Version: 0.9.5.post3 ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ https://mk1.ai ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ The license key for the current software has been verified as ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ belonging to: ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ Chai Research Corp. ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ║ ║
undi95-meta-llama-3-70b-6209-v21-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission trace2333-duduk-llama3-v1_v3: ('http://trace2333-duduk-llama3-v1-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"TypeError : SamplingParameters.__init__() got an unexpected keyword argument \'reward_max_tokens\'"}')
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Downloaded to shared memory in 292.504s
undi95-meta-llama-3-70b-6209-v21-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpwxs7v0zv, device:0
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Saving flywheel model at /dev/shm/model_cache
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Loading 0: 0%| | 0/723 [00:00<?, ?it/s] Loading 0: 0%| | 3/723 [00:01<04:01, 2.99it/s] Loading 0: 1%| | 4/723 [00:01<04:21, 2.75it/s] Loading 0: 1%| | 5/723 [00:01<04:22, 2.74it/s] Loading 0: 1%| | 8/723 [00:01<02:09, 5.54it/s] Loading 0: 1%|▏ | 10/723 [00:02<01:39, 7.15it/s] Loading 0: 2%|▏ | 12/723 [00:02<01:57, 6.05it/s] Loading 0: 2%|▏ | 13/723 [00:02<01:51, 6.38it/s] Loading 0: 2%|▏ | 14/723 [00:02<01:45, 6.69it/s] Loading 0: 2%|▏ | 16/723 [00:02<01:30, 7.80it/s] Loading 0: 2%|▏ | 17/723 [00:03<02:14, 5.23it/s] Loading 0: 2%|▏ | 18/723 [00:03<02:54, 4.05it/s] Loading 0: 3%|▎ | 21/723 [00:04<02:14, 5.23it/s] Loading 0: 3%|▎ | 22/723 [00:04<02:41, 4.35it/s] Loading 0: 3%|▎ | 23/723 [00:04<03:06, 3.76it/s] Loading 0: 4%|▎ | 26/723 [00:05<01:50, 6.29it/s] Loading 0: 4%|▍ | 28/723 [00:05<01:29, 7.75it/s] Loading 0: 4%|▍ | 30/723 [00:05<01:43, 6.69it/s] Loading 0: 4%|▍ | 31/723 [00:05<02:12, 5.22it/s] Loading 0: 4%|▍ | 32/723 [00:06<03:02, 3.78it/s] Loading 0: 5%|▍ | 35/723 [00:06<02:06, 5.44it/s] Loading 0: 5%|▍ | 36/723 [00:07<02:11, 5.21it/s] Loading 0: 5%|▌ | 39/723 [00:07<01:44, 6.57it/s] Loading 0: 6%|▌ | 40/723 [00:07<01:55, 5.90it/s] Loading 0: 6%|▌ | 42/723 [00:07<01:36, 7.08it/s] Loading 0: 6%|▌ | 43/723 [00:08<03:18, 3.43it/s] Loading 0: 6%|▌ | 44/723 [00:09<04:52, 2.33it/s] Loading 0: 6%|▌ | 45/723 [00:10<06:04, 1.86it/s] Loading 0: 7%|▋ | 48/723 [00:11<04:38, 2.42it/s] Loading 0: 7%|▋ | 49/723 [00:12<05:31, 2.03it/s] Loading 0: 7%|▋ | 50/723 [00:13<06:21, 1.76it/s] Loading 0: 7%|▋ | 53/723 [00:13<03:48, 2.94it/s] Loading 0: 7%|▋ | 54/723 [00:13<03:38, 3.06it/s] Loading 0: 8%|▊ | 57/723 [00:14<03:30, 3.17it/s] Loading 0: 8%|▊ | 58/723 [00:15<04:33, 2.43it/s] Loading 0: 8%|▊ | 59/723 [00:16<05:05, 2.17it/s] Loading 0: 9%|▊ | 62/723 [00:16<02:58, 3.71it/s] Loading 0: 9%|▊ | 63/723 [00:16<02:40, 4.12it/s] Loading 0: 9%|▉ | 66/723 [00:16<01:44, 6.29it/s] Loading 0: 9%|▉ | 68/723 [00:16<01:29, 7.36it/s] Loading 0: 10%|▉ | 70/723 [00:17<02:29, 4.38it/s] Loading 0: 10%|▉ | 71/723 [00:17<02:52, 3.79it/s] Loading 0: 10%|█ | 73/723 [00:18<02:07, 5.08it/s] Loading 0: 10%|█ | 75/723 [00:18<02:12, 4.89it/s] Loading 0: 11%|█ | 76/723 [00:18<02:38, 4.08it/s] Loading 0: 11%|█ | 77/723 [00:19<03:01, 3.56it/s] Loading 0: 11%|█ | 80/723 [00:19<01:49, 5.90it/s] Loading 0: 11%|█▏ | 82/723 [00:19<01:28, 7.26it/s] Loading 0: 12%|█▏ | 84/723 [00:20<01:41, 6.31it/s] Loading 0: 12%|█▏ | 85/723 [00:20<02:10, 4.90it/s] Loading 0: 12%|█▏ | 86/723 [00:20<02:35, 4.10it/s] Loading 0: 12%|█▏ | 89/723 [00:20<01:35, 6.61it/s] Loading 0: 13%|█▎ | 91/723 [00:21<01:18, 8.02it/s] Loading 0: 13%|█▎ | 93/723 [00:21<01:52, 5.59it/s] Loading 0: 13%|█▎ | 94/723 [00:22<02:22, 4.43it/s] Loading 0: 13%|█▎ | 95/723 [00:22<02:48, 3.73it/s] Loading 0: 14%|█▎ | 98/723 [00:22<01:42, 6.09it/s] Loading 0: 14%|█▍ | 100/723 [00:22<01:23, 7.43it/s] Loading 0: 14%|█▍ | 102/723 [00:23<01:36, 6.40it/s] Loading 0: 14%|█▍ | 103/723 [00:23<02:05, 4.95it/s] Loading 0: 14%|█▍ | 104/723 [00:24<02:28, 4.16it/s] Loading 0: 15%|█▍ | 107/723 [00:24<01:31, 6.72it/s] Loading 0: 15%|█▌ | 109/723 [00:24<01:15, 8.15it/s] Loading 0: 15%|█▌ | 111/723 [00:25<02:06, 4.84it/s] Loading 0: 16%|█▌ | 113/723 [00:25<01:40, 6.10it/s] Loading 0: 16%|█▌ | 115/723 [00:25<01:21, 7.44it/s] Loading 0: 16%|█▌ | 117/723 [00:25<01:46, 5.69it/s] Loading 0: 17%|█▋ | 120/723 [00:26<01:36, 6.25it/s] Loading 0: 17%|█▋ | 121/723 [00:26<01:46, 5.66it/s] Loading 0: 17%|█▋ | 123/723 [00:26<01:22, 7.27it/s] Loading 0: 17%|█▋ | 124/723 [00:40<01:22, 7.27it/s] Loading 0: 17%|█▋ | 125/723 [00:47<33:15, 3.34s/it] Loading 0: 18%|█▊ | 130/723 [00:47<15:53, 1.61s/it] Loading 0: 19%|█▊ | 135/723 [00:48<09:07, 1.07it/s] Loading 0: 19%|█▉ | 140/723 [00:48<05:42, 1.70it/s] Loading 0: 20%|█▉ | 143/723 [00:48<04:29, 2.15it/s] Loading 0: 20%|██ | 147/723 [00:48<03:08, 3.05it/s] Loading 0: 21%|██ | 150/723 [00:48<02:27, 3.89it/s] Loading 0: 22%|██▏ | 156/723 [00:48<01:28, 6.37it/s] Loading 0: 22%|██▏ | 160/723 [00:49<01:09, 8.12it/s] Loading 0: 23%|██▎ | 168/723 [00:49<00:46, 11.96it/s] Loading 0: 24%|██▎ | 171/723 [00:49<00:45, 12.25it/s] Loading 0: 24%|██▍ | 175/723 [00:49<00:38, 14.34it/s] Loading 0: 25%|██▍ | 179/723 [00:49<00:31, 17.42it/s] Loading 0: 25%|██▌ | 184/723 [00:49<00:26, 20.26it/s] Loading 0: 26%|██▌ | 189/723 [00:50<00:21, 24.31it/s] Loading 0: 27%|██▋ | 194/723 [00:50<00:22, 24.03it/s] Loading 0: 27%|██▋ | 198/723 [00:50<00:23, 22.29it/s] Loading 0: 28%|██▊ | 202/723 [00:50<00:22, 22.99it/s] Loading 0: 29%|██▊ | 207/723 [00:50<00:19, 27.02it/s] Loading 0: 29%|██▉ | 211/723 [00:50<00:18, 27.31it/s] Loading 0: 30%|██▉ | 216/723 [00:51<00:16, 30.94it/s] Loading 0: 30%|███ | 220/723 [00:51<00:23, 21.44it/s] Loading 0: 31%|███ | 224/723 [00:51<00:20, 24.58it/s] Loading 0: 32%|███▏ | 229/723 [00:51<00:18, 26.15it/s] Loading 0: 32%|███▏ | 234/723 [00:51<00:16, 29.78it/s] Loading 0: 33%|███▎ | 238/723 [00:51<00:16, 28.83it/s] Loading 0: 33%|███▎ | 242/723 [00:52<00:19, 24.56it/s] Loading 0: 34%|███▍ | 246/723 [00:52<00:18, 25.54it/s] Loading 0: 34%|███▍ | 249/723 [00:52<00:19, 24.00it/s] Loading 0: 35%|███▌ | 255/723 [00:52<00:15, 30.62it/s] Loading 0: 36%|███▌ | 259/723 [00:52<00:15, 29.44it/s] Loading 0: 36%|███▋ | 263/723 [00:52<00:14, 30.79it/s] Loading 0: 37%|███▋ | 268/723 [00:53<00:17, 26.74it/s] Loading 0: 37%|███▋ | 268/723 [01:14<00:17, 26.74it/s] Loading 0: 37%|███▋ | 269/723 [01:14<14:13, 1.88s/it] Loading 0: 38%|███▊ | 273/723 [01:14<09:29, 1.27s/it] Loading 0: 38%|███▊ | 276/723 [01:14<07:02, 1.06it/s] Loading 0: 39%|███▉ | 282/723 [01:14<04:01, 1.82it/s] Loading 0: 40%|███▉ | 286/723 [01:14<02:55, 2.50it/s] Loading 0: 41%|████ | 294/723 [01:15<01:41, 4.23it/s] Loading 0: 41%|████ | 297/723 [01:15<01:27, 4.88it/s] Loading 0: 42%|████▏ | 301/723 [01:15<01:07, 6.27it/s] Loading 0: 42%|████▏ | 305/723 [01:15<00:51, 8.19it/s] Loading 0: 43%|████▎ | 310/723 [01:15<00:38, 10.78it/s] Loading 0: 43%|████▎ | 314/723 [01:15<00:30, 13.47it/s] Loading 0: 44%|████▍ | 320/723 [01:16<00:25, 16.06it/s] Loading 0: 45%|████▍ | 323/723 [01:16<00:25, 15.46it/s] Loading 0: 45%|████▌ | 328/723 [01:16<00:21, 18.39it/s] Loading 0: 46%|████▌ | 332/723 [01:16<00:18, 21.49it/s] Loading 0: 47%|████▋ | 337/723 [01:16<00:16, 23.46it/s] Loading 0: 47%|████▋ | 341/723 [01:16<00:14, 26.22it/s] Loading 0: 48%|████▊ | 345/723 [01:17<00:18, 20.67it/s] Loading 0: 48%|████▊ | 348/723 [01:17<00:18, 20.40it/s] Loading 0: 49%|████▉ | 354/723 [01:17<00:13, 26.49it/s] Loading 0: 50%|████▉ | 358/723 [01:17<00:14, 26.07it/s] Loading 0: 50%|█████ | 362/723 [01:17<00:13, 27.56it/s] Loading 0: 51%|█████ | 366/723 [01:17<00:12, 28.68it/s] Loading 0: 51%|█████ | 370/723 [01:18<00:16, 21.73it/s] Loading 0: 52%|█████▏ | 373/723 [01:18<00:16, 21.52it/s] Loading 0: 52%|█████▏ | 377/723 [01:18<00:13, 24.98it/s] Loading 0: 53%|█████▎ | 382/723 [01:18<00:12, 26.36it/s] Loading 0: 53%|█████▎ | 386/723 [01:18<00:11, 29.10it/s] Loading 0: 54%|█████▍ | 390/723 [01:18<00:10, 31.50it/s] Loading 0: 54%|█████▍ | 394/723 [01:18<00:13, 24.93it/s] Loading 0: 55%|█████▍ | 397/723 [01:19<00:13, 23.49it/s] Loading 0: 55%|█████▌ | 400/723 [01:19<00:14, 22.74it/s] Loading 0: 55%|█████▌ | 400/723 [01:34<00:14, 22.74it/s] Loading 0: 55%|█████▌ | 401/723 [01:40<12:20, 2.30s/it] Loading 0: 56%|█████▋ | 408/723 [01:40<05:50, 1.11s/it] Loading 0: 57%|█████▋ | 412/723 [01:40<04:06, 1.26it/s] Loading 0: 58%|█████▊ | 420/723 [01:40<02:14, 2.25it/s] Loading 0: 59%|█████▊ | 424/723 [01:40<01:44, 2.86it/s] Loading 0: 59%|█████▉ | 427/723 [01:41<01:24, 3.50it/s] Loading 0: 60%|█████▉ | 432/723 [01:41<00:57, 5.03it/s] Loading 0: 60%|██████ | 436/723 [01:41<00:44, 6.48it/s] Loading 0: 61%|██████ | 440/723 [01:41<00:33, 8.49it/s] Loading 0: 62%|██████▏ | 446/723 [01:41<00:24, 11.15it/s] Loading 0: 62%|██████▏ | 449/723 [01:41<00:23, 11.56it/s] Loading 0: 63%|██████▎ | 454/723 [01:42<00:18, 14.48it/s] Loading 0: 63%|██████▎ | 458/723 [01:42<00:15, 17.58it/s] Loading 0: 64%|██████▍ | 463/723 [01:42<00:12, 20.30it/s] Loading 0: 65%|██████▍ | 467/723 [01:42<00:10, 23.29it/s] Loading 0: 65%|██████▌ | 471/723 [01:42<00:13, 18.97it/s] Loading 0: 66%|██████▌ | 474/723 [01:42<00:13, 19.02it/s] Loading 0: 66%|██████▋ | 480/723 [01:43<00:09, 25.22it/s] Loading 0: 67%|██████▋ | 484/723 [01:43<00:09, 25.22it/s] Loading 0: 67%|██████▋ | 488/723 [01:43<00:08, 27.14it/s] Loading 0: 68%|██████▊ | 492/723 [01:43<00:08, 28.49it/s] Loading 0: 69%|██████▊ | 496/723 [01:43<00:10, 22.36it/s] Loading 0: 69%|██████▉ | 499/723 [01:43<00:10, 21.87it/s] Loading 0: 70%|██████▉ | 503/723 [01:43<00:08, 25.45it/s] Loading 0: 70%|███████ | 508/723 [01:44<00:08, 26.77it/s] Loading 0: 71%|███████ | 512/723 [01:44<00:07, 29.38it/s] Loading 0: 71%|███████▏ | 516/723 [01:44<00:06, 31.56it/s] Loading 0: 72%|███████▏ | 520/723 [01:44<00:08, 25.27it/s] Loading 0: 72%|███████▏ | 523/723 [01:44<00:08, 23.77it/s] Loading 0: 73%|███████▎ | 526/723 [01:44<00:08, 22.95it/s] Loading 0: 73%|███████▎ | 530/723 [01:44<00:07, 26.64it/s] Loading 0: 74%|███████▍ | 535/723 [01:45<00:06, 27.69it/s] Loading 0: 74%|███████▍ | 535/723 [02:05<00:06, 27.69it/s] Loading 0: 74%|███████▍ | 536/723 [02:05<06:14, 2.00s/it] Loading 0: 76%|███████▌ | 546/723 [02:06<02:33, 1.16it/s] Loading 0: 76%|███████▌ | 550/723 [02:06<01:55, 1.50it/s] Loading 0: 76%|███████▋ | 553/723 [02:06<01:31, 1.87it/s] Loading 0: 77%|███████▋ | 557/723 [02:06<01:04, 2.57it/s] Loading 0: 78%|███████▊ | 562/723 [02:06<00:43, 3.71it/s] Loading 0: 78%|███████▊ | 566/723 [02:06<00:31, 4.96it/s] Loading 0: 79%|███████▉ | 572/723 [02:07<00:21, 7.06it/s] Loading 0: 80%|███████▉ | 575/723 [02:07<00:18, 7.86it/s] Loading 0: 80%|████████ | 580/723 [02:07<00:13, 10.41it/s] Loading 0: 81%|████████ | 584/723 [02:07<00:10, 13.09it/s] Loading 0: 81%|████████▏ | 589/723 [02:07<00:08, 15.98it/s] Loading 0: 82%|████████▏ | 593/723 [02:07<00:06, 19.12it/s] Loading 0: 83%|████████▎ | 597/723 [02:08<00:07, 17.14it/s] Loading 0: 83%|████████▎ | 600/723 [02:08<00:06, 17.72it/s] Loading 0: 84%|████████▍ | 606/723 [02:08<00:04, 23.92it/s] Loading 0: 84%|████████▍ | 610/723 [02:08<00:04, 24.40it/s] Loading 0: 85%|████████▍ | 614/723 [02:08<00:04, 26.29it/s] Loading 0: 85%|████████▌ | 618/723 [02:08<00:03, 28.01it/s] Loading 0: 86%|████████▌ | 622/723 [02:09<00:04, 22.50it/s] Loading 0: 86%|████████▋ | 625/723 [02:09<00:04, 22.05it/s] Loading 0: 87%|████████▋ | 630/723 [02:09<00:03, 26.56it/s] Loading 0: 88%|████████▊ | 634/723 [02:09<00:03, 26.81it/s] Loading 0: 88%|████████▊ | 639/723 [02:09<00:02, 30.53it/s] Loading 0: 89%|████████▉ | 644/723 [02:09<00:02, 33.47it/s] Loading 0: 90%|████████▉ | 648/723 [02:10<00:03, 22.60it/s] Loading 0: 90%|█████████ | 652/723 [02:10<00:02, 23.69it/s] Loading 0: 91%|█████████ | 656/723 [02:10<00:02, 26.73it/s] Loading 0: 91%|█████████▏| 661/723 [02:10<00:02, 27.55it/s] Loading 0: 92%|█████████▏| 665/723 [02:10<00:01, 30.03it/s] Loading 0: 93%|█████████▎| 672/723 [02:10<00:01, 28.81it/s] Loading 0: 93%|█████████▎| 674/723 [02:25<00:01, 28.81it/s] Loading 0: 93%|█████████▎| 675/723 [02:31<01:09, 1.44s/it] Loading 0: 94%|█████████▍| 679/723 [02:32<00:46, 1.05s/it] Loading 0: 94%|█████████▍| 683/723 [02:32<00:30, 1.31it/s] Loading 0: 95%|█████████▌| 688/723 [02:32<00:18, 1.93it/s] Loading 0: 96%|█████████▌| 692/723 [02:32<00:11, 2.63it/s] Loading 0: 97%|█████████▋| 698/723 [02:32<00:06, 3.90it/s] Loading 0: 97%|█████████▋| 702/723 [02:32<00:04, 4.91it/s] Loading 0: 98%|█████████▊| 706/723 [02:33<00:02, 6.27it/s] Loading 0: 98%|█████████▊| 710/723 [02:33<00:01, 8.21it/s] Loading 0: 99%|█████████▉| 715/723 [02:33<00:00, 10.83it/s] Loading 0: 99%|█████████▉| 719/723 [02:33<00:00, 13.54it/s] Loading 0: 100%|█████████▉| 722/723 [02:45<00:00, 13.54it/s] Loading 0: 100%|██████████| 723/723 [02:45<00:00, 1.15it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
undi95-meta-llama-3-70b-6209-v21-mkmlizer: quantized model in 187.411s
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Processed model Undi95/Meta-Llama-3-70B-Instruct-hf in 479.915s
undi95-meta-llama-3-70b-6209-v21-mkmlizer: creating bucket guanaco-mkml-models
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
undi95-meta-llama-3-70b-6209-v21-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/config.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/special_tokens_map.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/tokenizer_config.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/tokenizer.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.5.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/flywheel_model.5.safetensors
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/flywheel_model.0.safetensors
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/flywheel_model.1.safetensors
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/flywheel_model.2.safetensors
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/flywheel_model.3.safetensors
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /dev/shm/model_cache/flywheel_model.4.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v21/flywheel_model.4.safetensors
undi95-meta-llama-3-70b-6209-v21-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
undi95-meta-llama-3-70b-6209-v21-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:950: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v21-mkmlizer: warnings.warn(
undi95-meta-llama-3-70b-6209-v21-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v21-mkmlizer: warnings.warn(
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 2.38it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.86it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.53it/s]
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Saving duration: 1.400s
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 13.486s
undi95-meta-llama-3-70b-6209-v21-mkmlizer: creating bucket guanaco-reward-models
undi95-meta-llama-3-70b-6209-v21-mkmlizer: Bucket 's3://guanaco-reward-models/' created
undi95-meta-llama-3-70b-6209-v21-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v21_reward
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v21_reward/config.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v21_reward/special_tokens_map.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v21_reward/tokenizer_config.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v21_reward/merges.txt
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v21_reward/vocab.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v21_reward/tokenizer.json
undi95-meta-llama-3-70b-6209-v21-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v21_reward/reward.tensors
Job undi95-meta-llama-3-70b-6209-v21-mkmlizer completed after 555.61s with status: succeeded
Stopping job with name undi95-meta-llama-3-70b-6209-v21-mkmlizer
Pipeline stage MKMLizer completed in 556.54s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.12s
Running pipeline stage ISVCDeployer
Creating inference service undi95-meta-llama-3-70b-6209-v21
Waiting for inference service undi95-meta-llama-3-70b-6209-v21 to be ready
Failed to get response for submission chaiml-sao10k-l3-rp-v3-3_v45: ('http://chaiml-sao10k-l3-rp-v3-3-v45-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"TypeError : SamplingParameters.__init__() got an unexpected keyword argument \'reward_max_tokens\'"}')
Inference service undi95-meta-llama-3-70b-6209-v21 ready after 111.2292127609253s
Pipeline stage ISVCDeployer completed in 112.87s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.8201165199279785s
Received healthy response to inference request in 3.7118446826934814s
Received healthy response to inference request in 3.765044927597046s
Received healthy response to inference request in 3.8749005794525146s
Received healthy response to inference request in 3.758925199508667s
5 requests
0 failed requests
5th percentile: 3.7212607860565186
10th percentile: 3.7306768894195557
20th percentile: 3.74950909614563
30th percentile: 3.7601491451263427
40th percentile: 3.7625970363616945
50th percentile: 3.765044927597046
60th percentile: 3.808987188339233
70th percentile: 3.852929449081421
80th percentile: 4.063943767547608
90th percentile: 4.442030143737793
95th percentile: 4.631073331832885
99th percentile: 4.7823078823089595
mean time: 3.9861663818359374
Pipeline stage StressChecker completed in 20.73s
undi95-meta-llama-3-70b_6209_v21 status is now deployed due to DeploymentManager action
undi95-meta-llama-3-70b_6209_v21 status is now inactive due to auto deactivation removed underperforming models
undi95-meta-llama-3-70b_6209_v21 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics