submission_id: undi95-meta-llama-3-70b_6209_v17
developer_uid: chai_backend_admin
alignment_samples: 0
best_of: 2
celo_rating: 1185.37
display_name: undi95-meta-llama-3-70b_6209_v17
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s Persona: {memory}<|im_end|>\n", 'prompt_template': '<|im_start|>system\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 50, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|im_end|>', '<|im_start|>', '\n\n'], 'max_input_tokens': 512, 'best_of': 2, 'max_output_tokens': 64}
is_internal_developer: True
language_model: Undi95/Meta-Llama-3-70B-Instruct-hf
max_input_tokens: 512
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: Undi95/Meta-Llama-3-70B-
model_name: undi95-meta-llama-3-70b_6209_v17
model_num_parameters: 70553706496.0
model_repo: Undi95/Meta-Llama-3-70B-Instruct-hf
model_size: 71B
num_battles: 5132
num_wins: 2518
propriety_score: 0.7280858676207513
propriety_total_count: 1118.0
ranking_group: single
reward_formatter: {'bot_template': 'Bot: {message}\n', 'memory_template': 'Memory: {memory}\n', 'prompt_template': '{prompt}\n', 'response_template': 'Bot:', 'truncate_by_message': False, 'user_template': 'User: {message}\n'}
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
status: torndown
submission_type: basic
timestamp: 2024-07-18T23:46:14+00:00
us_pacific_date: 2024-07-18
win_ratio: 0.4906469212782541
Resubmit model
Running pipeline stage MKMLizer
Starting job with name undi95-meta-llama-3-70b-6209-v17-mkmlizer
Waiting for job on undi95-meta-llama-3-70b-6209-v17-mkmlizer to finish
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ _____ __ __ ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ /___/ ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ Version: 0.9.5.post3 ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ https://mk1.ai ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ The license key for the current software has been verified as ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ belonging to: ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ Chai Research Corp. ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ║ ║
undi95-meta-llama-3-70b-6209-v17-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Downloaded to shared memory in 312.848s
undi95-meta-llama-3-70b-6209-v17-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpekg5tt1s, device:0
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Saving flywheel model at /dev/shm/model_cache
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Loading 0: 0%| | 0/723 [00:00<?, ?it/s] Loading 0: 1%| | 4/723 [00:00<00:24, 28.79it/s] Loading 0: 1%| | 9/723 [00:00<00:19, 36.65it/s] Loading 0: 2%|▏ | 14/723 [00:00<00:17, 39.62it/s] Loading 0: 3%|▎ | 19/723 [00:00<00:29, 23.79it/s] Loading 0: 3%|▎ | 23/723 [00:00<00:31, 22.22it/s] Loading 0: 4%|▍ | 30/723 [00:01<00:22, 30.76it/s] Loading 0: 5%|▍ | 34/723 [00:01<00:23, 29.78it/s] Loading 0: 6%|▌ | 42/723 [00:01<00:22, 30.13it/s] Loading 0: 6%|▋ | 46/723 [00:01<00:25, 26.51it/s] Loading 0: 7%|▋ | 49/723 [00:01<00:26, 25.50it/s] Loading 0: 7%|▋ | 54/723 [00:01<00:22, 29.45it/s] Loading 0: 8%|▊ | 58/723 [00:02<00:22, 29.24it/s] Loading 0: 9%|▊ | 63/723 [00:02<00:20, 32.74it/s] Loading 0: 9%|▉ | 68/723 [00:02<00:22, 29.37it/s] Loading 0: 10%|▉ | 72/723 [00:02<00:25, 25.40it/s] Loading 0: 11%|█ | 76/723 [00:02<00:25, 25.43it/s] Loading 0: 11%|█ | 81/723 [00:02<00:21, 29.41it/s] Loading 0: 12%|█▏ | 85/723 [00:02<00:21, 29.25it/s] Loading 0: 12%|█▏ | 90/723 [00:03<00:19, 32.78it/s] Loading 0: 13%|█▎ | 94/723 [00:03<00:27, 22.76it/s] Loading 0: 14%|█▎ | 99/723 [00:03<00:23, 26.97it/s] Loading 0: 14%|█▍ | 103/723 [00:03<00:22, 27.42it/s] Loading 0: 15%|█▍ | 108/723 [00:03<00:19, 31.21it/s] Loading 0: 15%|█▌ | 112/723 [00:03<00:20, 30.12it/s] Loading 0: 16%|█▌ | 116/723 [00:04<00:23, 25.82it/s] Loading 0: 17%|█▋ | 120/723 [00:04<00:22, 26.69it/s] Loading 0: 17%|█▋ | 123/723 [00:04<00:23, 25.13it/s] Loading 0: 17%|█▋ | 126/723 [00:19<12:57, 1.30s/it] Loading 0: 18%|█▊ | 130/723 [00:19<08:49, 1.12it/s] Loading 0: 19%|█▊ | 134/723 [00:19<06:04, 1.62it/s] Loading 0: 19%|█▉ | 138/723 [00:19<04:14, 2.30it/s] Loading 0: 20%|█▉ | 142/723 [00:20<03:05, 3.14it/s] Loading 0: 20%|██ | 145/723 [00:20<02:25, 3.97it/s] Loading 0: 20%|██ | 148/723 [00:20<01:54, 5.03it/s] Loading 0: 21%|██ | 152/723 [00:20<01:20, 7.09it/s] Loading 0: 22%|██▏ | 157/723 [00:20<00:57, 9.90it/s] Loading 0: 22%|██▏ | 162/723 [00:20<00:41, 13.47it/s] Loading 0: 23%|██▎ | 168/723 [00:21<00:34, 16.26it/s] Loading 0: 24%|██▍ | 172/723 [00:21<00:32, 16.79it/s] Loading 0: 24%|██▍ | 175/723 [00:21<00:31, 17.51it/s] Loading 0: 25%|██▍ | 179/723 [00:21<00:25, 20.96it/s] Loading 0: 25%|██▌ | 184/723 [00:21<00:23, 23.32it/s] Loading 0: 26%|██▌ | 189/723 [00:21<00:19, 27.28it/s] Loading 0: 27%|██▋ | 194/723 [00:22<00:20, 25.45it/s] Loading 0: 27%|██▋ | 197/723 [00:22<00:24, 21.45it/s] Loading 0: 28%|██▊ | 202/723 [00:22<00:22, 23.57it/s] Loading 0: 28%|██▊ | 206/723 [00:22<00:19, 26.59it/s] Loading 0: 29%|██▉ | 211/723 [00:22<00:18, 27.64it/s] Loading 0: 30%|██▉ | 215/723 [00:22<00:16, 30.12it/s] Loading 0: 30%|███ | 219/723 [00:23<00:22, 22.27it/s] Loading 0: 31%|███ | 222/723 [00:23<00:23, 21.65it/s] Loading 0: 32%|███▏ | 228/723 [00:23<00:17, 28.01it/s] Loading 0: 32%|███▏ | 232/723 [00:23<00:17, 27.41it/s] Loading 0: 33%|███▎ | 236/723 [00:23<00:16, 29.15it/s] Loading 0: 33%|███▎ | 240/723 [00:23<00:15, 30.23it/s] Loading 0: 34%|███▎ | 244/723 [00:24<00:20, 23.54it/s] Loading 0: 34%|███▍ | 247/723 [00:24<00:20, 22.88it/s] Loading 0: 35%|███▍ | 251/723 [00:24<00:17, 26.34it/s] Loading 0: 35%|███▌ | 256/723 [00:24<00:17, 27.42it/s] Loading 0: 36%|███▌ | 260/723 [00:24<00:15, 30.14it/s] Loading 0: 37%|███▋ | 264/723 [00:24<00:14, 32.43it/s] Loading 0: 37%|███▋ | 268/723 [00:24<00:17, 26.03it/s] Loading 0: 37%|███▋ | 268/723 [00:40<00:17, 26.03it/s] Loading 0: 37%|███▋ | 269/723 [00:40<11:20, 1.50s/it] Loading 0: 38%|███▊ | 273/723 [00:40<07:22, 1.02it/s] Loading 0: 38%|███▊ | 276/723 [00:40<05:25, 1.37it/s] Loading 0: 39%|███▉ | 282/723 [00:40<03:03, 2.40it/s] Loading 0: 40%|███▉ | 286/723 [00:40<02:13, 3.28it/s] Loading 0: 41%|████ | 294/723 [00:41<01:18, 5.47it/s] Loading 0: 41%|████ | 297/723 [00:41<01:08, 6.20it/s] Loading 0: 42%|████▏ | 301/723 [00:41<00:53, 7.86it/s] Loading 0: 42%|████▏ | 306/723 [00:41<00:38, 10.70it/s] Loading 0: 43%|████▎ | 310/723 [00:41<00:32, 12.79it/s] Loading 0: 43%|████▎ | 314/723 [00:41<00:25, 15.80it/s] Loading 0: 44%|████▍ | 320/723 [00:42<00:22, 18.17it/s] Loading 0: 45%|████▍ | 324/723 [00:42<00:21, 18.34it/s] Loading 0: 45%|████▌ | 328/723 [00:42<00:19, 19.87it/s] Loading 0: 46%|████▌ | 333/723 [00:42<00:16, 24.17it/s] Loading 0: 47%|████▋ | 337/723 [00:42<00:15, 25.28it/s] Loading 0: 47%|████▋ | 342/723 [00:42<00:13, 29.13it/s] Loading 0: 48%|████▊ | 346/723 [00:43<00:17, 21.38it/s] Loading 0: 49%|████▊ | 351/723 [00:43<00:14, 25.62it/s] Loading 0: 49%|████▉ | 355/723 [00:43<00:13, 26.31it/s] Loading 0: 50%|████▉ | 360/723 [00:43<00:12, 30.01it/s] Loading 0: 50%|█████ | 364/723 [00:43<00:12, 29.01it/s] Loading 0: 51%|█████ | 368/723 [00:43<00:14, 25.13it/s] Loading 0: 51%|█████▏ | 372/723 [00:44<00:13, 26.15it/s] Loading 0: 52%|█████▏ | 375/723 [00:44<00:14, 24.71it/s] Loading 0: 53%|█████▎ | 381/723 [00:44<00:10, 31.49it/s] Loading 0: 53%|█████▎ | 385/723 [00:44<00:11, 30.33it/s] Loading 0: 54%|█████▍ | 389/723 [00:44<00:10, 31.95it/s] Loading 0: 54%|█████▍ | 394/723 [00:44<00:11, 28.11it/s] Loading 0: 55%|█████▌ | 398/723 [00:44<00:11, 27.91it/s] Loading 0: 55%|█████▌ | 401/723 [00:59<06:12, 1.16s/it] Loading 0: 56%|█████▋ | 408/723 [00:59<03:28, 1.51it/s] Loading 0: 57%|█████▋ | 412/723 [01:00<02:35, 2.00it/s] Loading 0: 58%|█████▊ | 420/723 [01:00<01:31, 3.32it/s] Loading 0: 59%|█████▊ | 424/723 [01:00<01:12, 4.12it/s] Loading 0: 59%|█████▉ | 427/723 [01:00<01:00, 4.92it/s] Loading 0: 60%|█████▉ | 432/723 [01:00<00:42, 6.89it/s] Loading 0: 60%|██████ | 436/723 [01:00<00:33, 8.66it/s] Loading 0: 61%|██████ | 441/723 [01:01<00:24, 11.68it/s] Loading 0: 62%|██████▏ | 446/723 [01:01<00:19, 13.88it/s] Loading 0: 62%|██████▏ | 450/723 [01:01<00:18, 14.95it/s] Loading 0: 63%|██████▎ | 454/723 [01:01<00:15, 16.85it/s] Loading 0: 63%|██████▎ | 459/723 [01:01<00:12, 21.11it/s] Loading 0: 64%|██████▍ | 463/723 [01:01<00:11, 22.75it/s] Loading 0: 65%|██████▍ | 468/723 [01:01<00:09, 26.94it/s] Loading 0: 65%|██████▌ | 472/723 [01:02<00:13, 18.36it/s] Loading 0: 66%|██████▌ | 477/723 [01:02<00:10, 22.57it/s] Loading 0: 67%|██████▋ | 481/723 [01:02<00:10, 23.95it/s] Loading 0: 67%|██████▋ | 486/723 [01:02<00:08, 27.92it/s] Loading 0: 68%|██████▊ | 490/723 [01:02<00:08, 27.69it/s] Loading 0: 68%|██████▊ | 494/723 [01:03<00:09, 24.23it/s] Loading 0: 69%|██████▉ | 498/723 [01:03<00:08, 25.26it/s] Loading 0: 69%|██████▉ | 501/723 [01:03<00:09, 24.11it/s] Loading 0: 70%|███████ | 507/723 [01:03<00:06, 30.89it/s] Loading 0: 71%|███████ | 511/723 [01:03<00:07, 29.78it/s] Loading 0: 71%|███████ | 515/723 [01:03<00:06, 31.33it/s] Loading 0: 72%|███████▏ | 520/723 [01:04<00:07, 27.25it/s] Loading 0: 72%|███████▏ | 523/723 [01:04<00:07, 25.44it/s] Loading 0: 73%|███████▎ | 526/723 [01:04<00:08, 24.38it/s] Loading 0: 73%|███████▎ | 531/723 [01:04<00:06, 29.01it/s] Loading 0: 74%|███████▍ | 535/723 [01:04<00:06, 28.95it/s] Loading 0: 75%|███████▍ | 539/723 [01:19<03:23, 1.11s/it] Loading 0: 76%|███████▌ | 546/723 [01:19<01:55, 1.53it/s] Loading 0: 76%|███████▌ | 549/723 [01:19<01:32, 1.88it/s] Loading 0: 76%|███████▋ | 553/723 [01:19<01:06, 2.55it/s] Loading 0: 77%|███████▋ | 558/723 [01:19<00:44, 3.73it/s] Loading 0: 78%|███████▊ | 562/723 [01:20<00:32, 4.91it/s] Loading 0: 78%|███████▊ | 567/723 [01:20<00:22, 6.94it/s] Loading 0: 79%|███████▉ | 572/723 [01:20<00:16, 8.94it/s] Loading 0: 80%|███████▉ | 576/723 [01:20<00:14, 10.36it/s] Loading 0: 80%|████████ | 580/723 [01:20<00:11, 12.40it/s] Loading 0: 81%|████████ | 585/723 [01:20<00:08, 16.24it/s] Loading 0: 81%|████████▏ | 589/723 [01:20<00:07, 18.44it/s] Loading 0: 82%|████████▏ | 594/723 [01:21<00:05, 22.62it/s] Loading 0: 83%|████████▎ | 598/723 [01:21<00:06, 18.09it/s] Loading 0: 83%|████████▎ | 603/723 [01:21<00:05, 22.18it/s] Loading 0: 84%|████████▍ | 607/723 [01:21<00:04, 23.38it/s] Loading 0: 85%|████████▍ | 612/723 [01:21<00:04, 27.28it/s] Loading 0: 85%|████████▌ | 616/723 [01:21<00:03, 26.99it/s] Loading 0: 86%|████████▌ | 620/723 [01:22<00:04, 22.97it/s] Loading 0: 86%|████████▋ | 624/723 [01:22<00:04, 24.12it/s] Loading 0: 87%|████████▋ | 627/723 [01:22<00:04, 23.00it/s] Loading 0: 88%|████████▊ | 633/723 [01:22<00:03, 29.49it/s] Loading 0: 88%|████████▊ | 637/723 [01:22<00:03, 28.28it/s] Loading 0: 89%|████████▊ | 641/723 [01:22<00:02, 29.98it/s] Loading 0: 89%|████████▉ | 646/723 [01:23<00:02, 26.31it/s] Loading 0: 90%|████████▉ | 649/723 [01:23<00:03, 24.66it/s] Loading 0: 90%|█████████ | 652/723 [01:23<00:03, 23.52it/s] Loading 0: 91%|█████████ | 656/723 [01:23<00:02, 27.00it/s] Loading 0: 91%|█████████▏| 661/723 [01:23<00:02, 27.87it/s] Loading 0: 92%|█████████▏| 666/723 [01:23<00:01, 31.41it/s] Loading 0: 93%|█████████▎| 672/723 [01:24<00:01, 28.83it/s] Loading 0: 93%|█████████▎| 676/723 [01:39<00:48, 1.03s/it] Loading 0: 94%|█████████▍| 679/723 [01:39<00:36, 1.22it/s] Loading 0: 94%|█████████▍| 683/723 [01:39<00:23, 1.71it/s] Loading 0: 95%|█████████▌| 688/723 [01:40<00:13, 2.52it/s] Loading 0: 96%|█████████▌| 693/723 [01:40<00:08, 3.65it/s] Loading 0: 97%|█████████▋| 698/723 [01:40<00:05, 4.95it/s] Loading 0: 97%|█████████▋| 701/723 [01:40<00:03, 5.74it/s] Loading 0: 98%|█████████▊| 706/723 [01:40<00:02, 7.90it/s] Loading 0: 98%|█████████▊| 711/723 [01:40<00:01, 10.73it/s] Loading 0: 99%|█████████▉| 715/723 [01:41<00:00, 12.86it/s] Loading 0: 99%|█████████▉| 719/723 [01:41<00:00, 15.86it/s] Loading 0: 100%|█████████▉| 722/723 [01:51<00:00, 15.86it/s] Loading 0: 100%|██████████| 723/723 [01:51<00:00, 1.24it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
undi95-meta-llama-3-70b-6209-v17-mkmlizer: quantized model in 128.650s
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Processed model Undi95/Meta-Llama-3-70B-Instruct-hf in 441.497s
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
undi95-meta-llama-3-70b-6209-v17-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/config.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/special_tokens_map.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/tokenizer_config.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/tokenizer.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/flywheel_model.5.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/flywheel_model.5.safetensors
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/flywheel_model.2.safetensors
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/flywheel_model.3.safetensors
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/flywheel_model.0.safetensors
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/flywheel_model.4.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/flywheel_model.4.safetensors
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/undi95-meta-llama-3-70b-6209-v17/flywheel_model.1.safetensors
undi95-meta-llama-3-70b-6209-v17-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
undi95-meta-llama-3-70b-6209-v17-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:950: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v17-mkmlizer: warnings.warn(
undi95-meta-llama-3-70b-6209-v17-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:778: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v17-mkmlizer: warnings.warn(
undi95-meta-llama-3-70b-6209-v17-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
undi95-meta-llama-3-70b-6209-v17-mkmlizer: warnings.warn(
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Downloading shards: 0%| | 0/2 [00:00<?, ?it/s] Downloading shards: 50%|█████ | 1/2 [00:05<00:05, 5.40s/it] Downloading shards: 100%|██████████| 2/2 [00:08<00:00, 3.98s/it] Downloading shards: 100%|██████████| 2/2 [00:08<00:00, 4.20s/it]
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 2.39it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.92it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.58it/s]
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Saving duration: 1.381s
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 13.488s
undi95-meta-llama-3-70b-6209-v17-mkmlizer: creating bucket guanaco-reward-models
undi95-meta-llama-3-70b-6209-v17-mkmlizer: Bucket 's3://guanaco-reward-models/' created
undi95-meta-llama-3-70b-6209-v17-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v17_reward
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v17_reward/config.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v17_reward/tokenizer_config.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v17_reward/special_tokens_map.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v17_reward/merges.txt
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v17_reward/vocab.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v17_reward/tokenizer.json
undi95-meta-llama-3-70b-6209-v17-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/undi95-meta-llama-3-70b-6209-v17_reward/reward.tensors
Job undi95-meta-llama-3-70b-6209-v17-mkmlizer completed after 571.27s with status: succeeded
Stopping job with name undi95-meta-llama-3-70b-6209-v17-mkmlizer
Pipeline stage MKMLizer completed in 572.88s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.30s
Running pipeline stage ISVCDeployer
Creating inference service undi95-meta-llama-3-70b-6209-v17
Waiting for inference service undi95-meta-llama-3-70b-6209-v17 to be ready
Inference service undi95-meta-llama-3-70b-6209-v17 ready after 205.47017431259155s
Pipeline stage ISVCDeployer completed in 207.62s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 5.240909814834595s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 4.259753465652466s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 4.246101140975952s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 4.273311138153076s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 4.255446434020996s
5 requests
0 failed requests
5th percentile: 4.247970199584961
10th percentile: 4.24983925819397
20th percentile: 4.253577375411988
30th percentile: 4.25630784034729
40th percentile: 4.258030652999878
50th percentile: 4.259753465652466
60th percentile: 4.26517653465271
70th percentile: 4.270599603652954
80th percentile: 4.46683087348938
90th percentile: 4.8538703441619875
95th percentile: 5.047390079498291
99th percentile: 5.202205867767334
mean time: 4.455104398727417
Pipeline stage StressChecker completed in 24.78s
undi95-meta-llama-3-70b_6209_v17 status is now deployed due to DeploymentManager action
undi95-meta-llama-3-70b_6209_v17 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of undi95-meta-llama-3-70b_6209_v17
Running pipeline stage ISVCDeleter
Checking if service undi95-meta-llama-3-70b-6209-v17 is running
Tearing down inference service undi95-meta-llama-3-70b-6209-v17
Service undi95-meta-llama-3-70b-6209-v17 has been torndown
Pipeline stage ISVCDeleter completed in 5.45s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key undi95-meta-llama-3-70b-6209-v17/config.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/flywheel_model.1.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/flywheel_model.2.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/flywheel_model.3.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/flywheel_model.4.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/flywheel_model.5.safetensors from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/tokenizer.json from bucket guanaco-mkml-models
Deleting key undi95-meta-llama-3-70b-6209-v17/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key undi95-meta-llama-3-70b-6209-v17_reward/config.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v17_reward/merges.txt from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v17_reward/reward.tensors from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v17_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v17_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v17_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key undi95-meta-llama-3-70b-6209-v17_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 12.60s
undi95-meta-llama-3-70b_6209_v17 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics