submission_id: shuttleai-shuttle-2-5-e0-2_v1
developer_uid: Fizzarolli
formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': True}
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.066, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64, 'reward_max_token_input': 256}
model_name: shuttleai-shuttle-2-5-e0-2_v1
model_repo: shuttleai/shuttle-2.5-e0.2
reward_formatter: {'bot_template': '{bot_name}: {message}\\n', 'memory_template': "{bot_name}'s Persona: {memory}\\n####\\n", 'prompt_template': '{prompt}\\n<START>\\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\\n'}
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
status: torndown
timestamp: 2024-07-30T16:58:15+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name shuttleai-shuttle-2-5-e0-2-v1-mkmlizer
Waiting for job on shuttleai-shuttle-2-5-e0-2-v1-mkmlizer to finish
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ _____ __ __ ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ /___/ ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ Version: 0.9.7 ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ https://mk1.ai ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ The license key for the current software has been verified as ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ belonging to: ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ Chai Research Corp. ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ║ ║
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Downloaded to shared memory in 267.107s
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp7o26gfv8, device:0
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: quantized model in 137.193s
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Processed model shuttleai/shuttle-2.5-e0.2 in 404.299s
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: creating bucket guanaco-mkml-models
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/config.json
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/special_tokens_map.json
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/tokenizer_config.json
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/tokenizer.json
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.5.safetensors s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.5.safetensors
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.2.safetensors
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.3.safetensors
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.0.safetensors
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.1.safetensors
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.4.safetensors s3://guanaco-mkml-models/shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.4.safetensors
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Loading 0: 0%| | 0/723 [00:00<?, ?it/s] Loading 0: 1%| | 4/723 [00:00<00:23, 30.46it/s] Loading 0: 1%| | 8/723 [00:00<00:27, 26.34it/s] Loading 0: 2%|▏ | 11/723 [00:00<00:29, 24.28it/s] Loading 0: 2%|▏ | 15/723 [00:00<00:24, 28.72it/s] Loading 0: 2%|▏ | 18/723 [00:00<00:42, 16.40it/s] Loading 0: 3%|▎ | 22/723 [00:01<00:36, 19.19it/s] Loading 0: 3%|▎ | 25/723 [00:01<00:34, 19.96it/s] Loading 0: 4%|▍ | 28/723 [00:01<00:31, 21.84it/s] Loading 0: 4%|▍ | 31/723 [00:01<00:31, 21.96it/s] Loading 0: 5%|▍ | 34/723 [00:01<00:31, 22.14it/s] Loading 0: 5%|▌ | 37/723 [00:01<00:29, 23.63it/s] Loading 0: 6%|▌ | 41/723 [00:01<00:24, 27.50it/s] Loading 0: 6%|▌ | 44/723 [00:02<00:36, 18.82it/s] Loading 0: 7%|▋ | 47/723 [00:02<00:33, 20.07it/s] Loading 0: 7%|▋ | 50/723 [00:02<00:42, 15.89it/s] Loading 0: 8%|▊ | 55/723 [00:02<00:31, 20.98it/s] Loading 0: 8%|▊ | 58/723 [00:02<00:31, 20.81it/s] Loading 0: 8%|▊ | 61/723 [00:02<00:31, 21.22it/s] Loading 0: 9%|▉ | 64/723 [00:02<00:29, 22.70it/s] Loading 0: 9%|▉ | 68/723 [00:03<00:33, 19.71it/s] Loading 0: 10%|▉ | 71/723 [00:03<00:39, 16.53it/s] Loading 0: 10%|█ | 75/723 [00:03<00:31, 20.52it/s] Loading 0: 11%|█ | 78/723 [00:03<00:34, 18.61it/s] Loading 0: 11%|█▏ | 82/723 [00:03<00:29, 21.42it/s] Loading 0: 12%|█▏ | 85/723 [00:04<00:29, 21.63it/s] Loading 0: 12%|█▏ | 88/723 [00:04<00:29, 21.69it/s] Loading 0: 13%|█▎ | 91/723 [00:04<00:27, 22.83it/s] Loading 0: 13%|█▎ | 94/723 [00:04<00:37, 16.58it/s] Loading 0: 13%|█▎ | 96/723 [00:04<00:38, 16.32it/s] Loading 0: 14%|█▍ | 100/723 [00:04<00:31, 19.91it/s] Loading 0: 14%|█▍ | 103/723 [00:05<00:30, 20.20it/s] Loading 0: 15%|█▍ | 106/723 [00:05<00:30, 20.50it/s] Loading 0: 15%|█▌ | 109/723 [00:05<00:28, 21.91it/s] Loading 0: 15%|█▌ | 112/723 [00:05<00:32, 18.58it/s] Loading 0: 16%|█▌ | 115/723 [00:05<00:29, 20.29it/s] Loading 0: 16%|█▋ | 118/723 [00:05<00:34, 17.36it/s] Loading 0: 17%|█▋ | 121/723 [00:05<00:32, 18.66it/s] Loading 0: 17%|█▋ | 124/723 [00:06<00:30, 19.51it/s] Loading 0: 17%|█▋ | 124/723 [00:21<00:30, 19.51it/s] Loading 0: 17%|█▋ | 125/723 [00:21<19:05, 1.92s/it] Loading 0: 18%|█▊ | 129/723 [00:21<11:26, 1.16s/it] Loading 0: 18%|█▊ | 131/723 [00:21<09:00, 1.10it/s] Loading 0: 19%|█▉ | 136/723 [00:21<05:00, 1.96it/s] Loading 0: 19%|█▉ | 139/723 [00:21<03:41, 2.63it/s] Loading 0: 20%|█▉ | 142/723 [00:22<02:51, 3.39it/s] Loading 0: 20%|██ | 145/723 [00:22<02:11, 4.38it/s] Loading 0: 20%|██ | 148/723 [00:22<01:41, 5.68it/s] Loading 0: 21%|██ | 150/723 [00:22<01:26, 6.59it/s] Loading 0: 21%|██▏ | 154/723 [00:22<00:59, 9.50it/s] Loading 0: 22%|██▏ | 157/723 [00:22<00:49, 11.40it/s] Loading 0: 22%|██▏ | 160/723 [00:22<00:42, 13.30it/s] Loading 0: 23%|██▎ | 163/723 [00:23<00:35, 15.57it/s] Loading 0: 23%|██▎ | 167/723 [00:23<00:28, 19.53it/s] Loading 0: 24%|██▎ | 170/723 [00:23<00:35, 15.64it/s] Loading 0: 24%|██▍ | 173/723 [00:23<00:32, 17.11it/s] Loading 0: 24%|██▍ | 176/723 [00:23<00:37, 14.74it/s] Loading 0: 25%|██▌ | 181/723 [00:24<00:27, 19.70it/s] Loading 0: 25%|██▌ | 184/723 [00:24<00:27, 19.71it/s] Loading 0: 26%|██▌ | 187/723 [00:24<00:26, 20.02it/s] Loading 0: 26%|██▋ | 190/723 [00:24<00:24, 21.61it/s] Loading 0: 27%|██▋ | 193/723 [00:24<00:22, 23.29it/s] Loading 0: 27%|██▋ | 196/723 [00:24<00:30, 17.21it/s] Loading 0: 28%|██▊ | 199/723 [00:25<00:30, 17.35it/s] Loading 0: 28%|██▊ | 202/723 [00:25<00:28, 18.60it/s] Loading 0: 28%|██▊ | 205/723 [00:25<00:26, 19.59it/s] Loading 0: 29%|██▉ | 208/723 [00:25<00:24, 21.19it/s] Loading 0: 29%|██▉ | 211/723 [00:25<00:24, 21.08it/s] Loading 0: 30%|██▉ | 214/723 [00:25<00:23, 21.55it/s] Loading 0: 30%|███ | 217/723 [00:25<00:22, 22.62it/s] Loading 0: 30%|███ | 220/723 [00:26<00:30, 16.72it/s] Loading 0: 31%|███ | 222/723 [00:26<00:30, 16.47it/s] Loading 0: 31%|███▏ | 226/723 [00:26<00:24, 20.40it/s] Loading 0: 32%|███▏ | 229/723 [00:26<00:24, 20.31it/s] Loading 0: 32%|███▏ | 232/723 [00:26<00:23, 20.71it/s] Loading 0: 33%|███▎ | 235/723 [00:26<00:22, 21.96it/s] Loading 0: 33%|███▎ | 238/723 [00:26<00:26, 18.51it/s] Loading 0: 33%|███▎ | 241/723 [00:27<00:23, 20.38it/s] Loading 0: 34%|███▎ | 244/723 [00:27<00:26, 17.80it/s] Loading 0: 34%|███▍ | 247/723 [00:27<00:25, 18.96it/s] Loading 0: 35%|███▍ | 250/723 [00:27<00:23, 19.92it/s] Loading 0: 35%|███▍ | 253/723 [00:27<00:22, 21.30it/s] Loading 0: 35%|███▌ | 256/723 [00:27<00:22, 21.11it/s] Loading 0: 36%|███▌ | 259/723 [00:27<00:21, 21.28it/s] Loading 0: 36%|███▌ | 262/723 [00:28<00:20, 22.57it/s] Loading 0: 37%|███▋ | 265/723 [00:28<00:19, 23.84it/s] Loading 0: 37%|███▋ | 268/723 [00:28<00:24, 18.73it/s] Loading 0: 37%|███▋ | 268/723 [00:43<00:24, 18.73it/s] Loading 0: 37%|███▋ | 269/723 [00:43<14:15, 1.88s/it] Loading 0: 37%|███▋ | 270/723 [00:43<12:14, 1.62s/it] Loading 0: 38%|███▊ | 273/723 [00:43<07:37, 1.02s/it] Loading 0: 38%|███▊ | 275/723 [00:43<05:47, 1.29it/s] Loading 0: 39%|███▊ | 280/723 [00:43<02:59, 2.46it/s] Loading 0: 39%|███▉ | 283/723 [00:44<02:12, 3.33it/s] Loading 0: 40%|███▉ | 286/723 [00:44<01:38, 4.45it/s] Loading 0: 40%|███▉ | 289/723 [00:44<01:13, 5.92it/s] Loading 0: 41%|████ | 293/723 [00:44<00:50, 8.44it/s] Loading 0: 41%|████ | 296/723 [00:44<00:47, 8.99it/s] Loading 0: 41%|████▏ | 299/723 [00:44<00:38, 10.93it/s] Loading 0: 42%|████▏ | 302/723 [00:45<00:39, 10.68it/s] Loading 0: 42%|████▏ | 307/723 [00:45<00:27, 15.10it/s] Loading 0: 43%|████▎ | 310/723 [00:45<00:25, 16.12it/s] Loading 0: 43%|████▎ | 313/723 [00:45<00:23, 17.20it/s] Loading 0: 44%|████▎ | 316/723 [00:45<00:21, 19.17it/s] Loading 0: 44%|████▍ | 319/723 [00:45<00:19, 20.97it/s] Loading 0: 45%|████▍ | 322/723 [00:46<00:24, 16.05it/s] Loading 0: 45%|████▍ | 325/723 [00:46<00:24, 16.46it/s] Loading 0: 45%|████▌ | 328/723 [00:46<00:21, 18.12it/s] Loading 0: 46%|████▌ | 331/723 [00:46<00:20, 19.22it/s] Loading 0: 46%|████▌ | 334/723 [00:46<00:19, 20.27it/s] Loading 0: 47%|████▋ | 337/723 [00:46<00:18, 20.33it/s] Loading 0: 47%|████▋ | 340/723 [00:46<00:18, 21.02it/s] Loading 0: 47%|████▋ | 343/723 [00:47<00:16, 22.44it/s] Loading 0: 48%|████▊ | 346/723 [00:47<00:22, 16.76it/s] Loading 0: 48%|████▊ | 348/723 [00:47<00:23, 15.89it/s] Loading 0: 49%|████▊ | 352/723 [00:47<00:18, 19.93it/s] Loading 0: 49%|████▉ | 355/723 [00:47<00:18, 19.99it/s] Loading 0: 50%|████▉ | 358/723 [00:47<00:17, 20.73it/s] Loading 0: 50%|████▉ | 361/723 [00:47<00:16, 22.15it/s] Loading 0: 50%|█████ | 364/723 [00:48<00:19, 18.52it/s] Loading 0: 51%|█████ | 367/723 [00:48<00:17, 20.09it/s] Loading 0: 51%|█████ | 370/723 [00:48<00:19, 17.79it/s] Loading 0: 52%|█████▏ | 373/723 [00:48<00:18, 19.05it/s] Loading 0: 52%|█████▏ | 376/723 [00:48<00:17, 19.72it/s] Loading 0: 52%|█████▏ | 379/723 [00:48<00:16, 21.30it/s] Loading 0: 53%|█████▎ | 382/723 [00:49<00:16, 20.78it/s] Loading 0: 53%|█████▎ | 385/723 [00:49<00:16, 20.95it/s] Loading 0: 54%|█████▎ | 388/723 [00:49<00:14, 22.45it/s] Loading 0: 54%|█████▍ | 391/723 [00:49<00:13, 23.79it/s] Loading 0: 54%|█████▍ | 394/723 [00:49<00:17, 19.06it/s] Loading 0: 55%|█████▍ | 397/723 [00:49<00:18, 17.43it/s] Loading 0: 55%|█████▌ | 400/723 [00:50<00:17, 18.15it/s] Loading 0: 55%|█████▌ | 400/723 [01:04<00:17, 18.15it/s] Loading 0: 55%|█████▌ | 401/723 [01:04<10:02, 1.87s/it] Loading 0: 56%|█████▌ | 405/723 [01:04<05:52, 1.11s/it] Loading 0: 56%|█████▋ | 408/723 [01:04<04:08, 1.27it/s] Loading 0: 57%|█████▋ | 410/723 [01:05<03:17, 1.59it/s] Loading 0: 57%|█████▋ | 415/723 [01:05<01:50, 2.79it/s] Loading 0: 58%|█████▊ | 419/723 [01:05<01:15, 4.02it/s] Loading 0: 58%|█████▊ | 422/723 [01:05<01:02, 4.84it/s] Loading 0: 59%|█████▉ | 425/723 [01:05<00:48, 6.17it/s] Loading 0: 59%|█████▉ | 428/723 [01:06<00:41, 7.03it/s] Loading 0: 60%|█████▉ | 433/723 [01:06<00:27, 10.43it/s] Loading 0: 60%|██████ | 436/723 [01:06<00:23, 12.01it/s] Loading 0: 61%|██████ | 439/723 [01:06<00:20, 13.62it/s] Loading 0: 61%|██████ | 442/723 [01:06<00:17, 15.78it/s] Loading 0: 62%|██████▏ | 445/723 [01:06<00:15, 18.01it/s] Loading 0: 62%|██████▏ | 448/723 [01:07<00:18, 15.15it/s] Loading 0: 62%|██████▏ | 451/723 [01:07<00:17, 15.51it/s] Loading 0: 63%|██████▎ | 454/723 [01:07<00:15, 17.14it/s] Loading 0: 63%|██████▎ | 457/723 [01:07<00:14, 18.33it/s] Loading 0: 64%|██████▎ | 460/723 [01:07<00:13, 19.89it/s] Loading 0: 64%|██████▍ | 463/723 [01:07<00:12, 20.18it/s] Loading 0: 64%|██████▍ | 466/723 [01:07<00:12, 20.86it/s] Loading 0: 65%|██████▍ | 469/723 [01:07<00:11, 22.12it/s] Loading 0: 65%|██████▌ | 472/723 [01:08<00:15, 16.44it/s] Loading 0: 66%|██████▌ | 474/723 [01:08<00:15, 16.20it/s] Loading 0: 66%|██████▌ | 478/723 [01:08<00:12, 19.51it/s] Loading 0: 67%|██████▋ | 481/723 [01:08<00:12, 19.33it/s] Loading 0: 67%|██████▋ | 484/723 [01:08<00:11, 20.04it/s] Loading 0: 67%|██████▋ | 487/723 [01:08<00:10, 21.58it/s] Loading 0: 68%|██████▊ | 490/723 [01:09<00:12, 18.47it/s] Loading 0: 68%|██████▊ | 493/723 [01:09<00:11, 20.34it/s] Loading 0: 69%|██████▊ | 496/723 [01:09<00:12, 17.70it/s] Loading 0: 69%|██████▉ | 499/723 [01:09<00:11, 19.06it/s] Loading 0: 69%|██████▉ | 502/723 [01:09<00:11, 19.47it/s] Loading 0: 70%|██████▉ | 505/723 [01:09<00:10, 20.86it/s] Loading 0: 70%|███████ | 508/723 [01:10<00:10, 20.59it/s] Loading 0: 71%|███████ | 511/723 [01:10<00:09, 21.21it/s] Loading 0: 71%|███████ | 514/723 [01:10<00:09, 22.89it/s] Loading 0: 72%|███████▏ | 517/723 [01:10<00:08, 24.32it/s] Loading 0: 72%|███████▏ | 520/723 [01:10<00:10, 19.19it/s] Loading 0: 72%|███████▏ | 523/723 [01:10<00:11, 18.00it/s] Loading 0: 73%|███████▎ | 526/723 [01:10<00:10, 18.66it/s] Loading 0: 73%|███████▎ | 528/723 [01:11<00:10, 17.89it/s] Loading 0: 74%|███████▎ | 532/723 [01:11<00:08, 21.60it/s] Loading 0: 74%|███████▍ | 535/723 [01:11<00:08, 21.15it/s] Loading 0: 74%|███████▍ | 535/723 [01:25<00:08, 21.15it/s] Loading 0: 74%|███████▍ | 536/723 [01:25<05:36, 1.80s/it] Loading 0: 75%|███████▍ | 540/723 [01:25<03:16, 1.08s/it] Loading 0: 75%|███████▌ | 543/723 [01:26<02:17, 1.31it/s] Loading 0: 76%|███████▌ | 546/723 [01:26<01:38, 1.80it/s] Loading 0: 76%|███████▌ | 548/723 [01:26<01:18, 2.24it/s] Loading 0: 76%|███████▌ | 550/723 [01:26<01:01, 2.82it/s] Loading 0: 76%|███████▋ | 553/723 [01:26<00:42, 4.00it/s] Loading 0: 77%|███████▋ | 555/723 [01:26<00:34, 4.88it/s] Loading 0: 77%|███████▋ | 559/723 [01:26<00:21, 7.46it/s] Loading 0: 78%|███████▊ | 562/723 [01:27<00:17, 9.24it/s] Loading 0: 78%|███████▊ | 565/723 [01:27<00:14, 11.20it/s] Loading 0: 79%|███████▊ | 568/723 [01:27<00:11, 13.35it/s] Loading 0: 79%|███████▉ | 571/723 [01:27<00:09, 15.87it/s] Loading 0: 79%|███████▉ | 574/723 [01:27<00:11, 13.45it/s] Loading 0: 80%|███████▉ | 576/723 [01:27<00:10, 13.92it/s] Loading 0: 80%|████████ | 580/723 [01:28<00:08, 16.73it/s] Loading 0: 81%|████████ | 583/723 [01:28<00:07, 17.74it/s] Loading 0: 81%|████████ | 586/723 [01:28<00:07, 19.19it/s] Loading 0: 81%|████████▏ | 589/723 [01:28<00:06, 19.42it/s] Loading 0: 82%|████████▏ | 592/723 [01:28<00:06, 20.09it/s] Loading 0: 82%|████████▏ | 595/723 [01:28<00:06, 21.33it/s] Loading 0: 83%|████████▎ | 598/723 [01:29<00:07, 16.10it/s] Loading 0: 83%|████████▎ | 600/723 [01:29<00:07, 16.04it/s] Loading 0: 84%|████████▎ | 604/723 [01:29<00:05, 19.99it/s] Loading 0: 84%|████████▍ | 607/723 [01:29<00:05, 19.98it/s] Loading 0: 84%|████████▍ | 610/723 [01:29<00:05, 20.61it/s] Loading 0: 85%|████████▍ | 613/723 [01:29<00:05, 21.85it/s] Loading 0: 85%|████████▌ | 616/723 [01:29<00:05, 18.50it/s] Loading 0: 86%|████████▌ | 619/723 [01:30<00:05, 20.44it/s] Loading 0: 86%|████████▌ | 622/723 [01:30<00:05, 17.71it/s] Loading 0: 86%|████████▋ | 625/723 [01:30<00:05, 19.04it/s] Loading 0: 87%|████████▋ | 628/723 [01:30<00:04, 19.33it/s] Loading 0: 87%|████████▋ | 631/723 [01:30<00:04, 20.86it/s] Loading 0: 88%|████████▊ | 634/723 [01:30<00:04, 20.75it/s] Loading 0: 88%|████████▊ | 637/723 [01:30<00:04, 21.20it/s] Loading 0: 89%|████████▊ | 640/723 [01:31<00:03, 22.43it/s] Loading 0: 89%|████████▉ | 643/723 [01:31<00:03, 23.85it/s] Loading 0: 89%|████████▉ | 646/723 [01:31<00:04, 18.82it/s] Loading 0: 90%|████████▉ | 649/723 [01:31<00:04, 17.66it/s] Loading 0: 90%|█████████ | 652/723 [01:31<00:03, 18.43it/s] Loading 0: 90%|█████████ | 654/723 [01:31<00:03, 17.67it/s] Loading 0: 91%|█████████ | 658/723 [01:31<00:03, 21.04it/s] Loading 0: 91%|█████████▏| 661/723 [01:32<00:02, 20.80it/s] Loading 0: 92%|█████████▏| 664/723 [01:32<00:02, 20.73it/s] Loading 0: 92%|█████████▏| 667/723 [01:32<00:02, 21.90it/s] Loading 0: 93%|█████████▎| 671/723 [01:32<00:02, 25.16it/s] Loading 0: 93%|█████████▎| 674/723 [01:32<00:02, 18.34it/s] Loading 0: 93%|█████████▎| 674/723 [01:47<00:02, 18.34it/s] Loading 0: 93%|█████████▎| 675/723 [01:47<01:23, 1.75s/it] Loading 0: 94%|█████████▍| 678/723 [01:47<00:53, 1.19s/it] Loading 0: 94%|█████████▍| 680/723 [01:47<00:40, 1.07it/s] Loading 0: 95%|█████████▍| 685/723 [01:47<00:19, 1.95it/s] Loading 0: 95%|█████████▌| 688/723 [01:48<00:13, 2.63it/s] Loading 0: 96%|█████████▌| 691/723 [01:48<00:09, 3.51it/s] Loading 0: 96%|█████████▌| 694/723 [01:48<00:06, 4.70it/s] Loading 0: 96%|█████████▋| 697/723 [01:48<00:04, 6.22it/s] Loading 0: 97%|█████████▋| 700/723 [01:48<00:03, 7.00it/s] Loading 0: 97%|█████████▋| 702/723 [01:48<00:02, 7.92it/s] Loading 0: 98%|█████████▊| 705/723 [01:48<00:01, 10.34it/s] Loading 0: 98%|█████████▊| 708/723 [01:49<00:01, 11.53it/s] Loading 0: 98%|█████████▊| 712/723 [01:49<00:00, 15.09it/s] Loading 0: 99%|█████████▉| 715/723 [01:49<00:00, 16.27it/s] Loading 0: 99%|█████████▉| 718/723 [01:49<00:00, 17.40it/s] Loading 0: 100%|█████████▉| 721/723 [01:49<00:00, 19.44it/s] Loading 0: 100%|█████████▉| 722/723 [02:00<00:00, 19.44it/s] Loading 0: 100%|██████████| 723/723 [02:00<00:00, 1.18s/it] /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:957: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: warnings.warn(
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:785: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: warnings.warn(
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: warnings.warn(
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Downloading shards: 0%| | 0/2 [00:00<?, ?it/s] Downloading shards: 50%|█████ | 1/2 [00:05<00:05, 5.81s/it] Downloading shards: 100%|██████████| 2/2 [00:08<00:00, 4.14s/it] Downloading shards: 100%|██████████| 2/2 [00:08<00:00, 4.39s/it]
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 2.11it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.55it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.22it/s]
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Saving duration: 1.406s
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 14.085s
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: creating bucket guanaco-reward-models
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: Bucket 's3://guanaco-reward-models/' created
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/shuttleai-shuttle-2-5-e0-2-v1_reward
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/shuttleai-shuttle-2-5-e0-2-v1_reward/config.json
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/shuttleai-shuttle-2-5-e0-2-v1_reward/special_tokens_map.json
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/shuttleai-shuttle-2-5-e0-2-v1_reward/tokenizer_config.json
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/shuttleai-shuttle-2-5-e0-2-v1_reward/merges.txt
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/shuttleai-shuttle-2-5-e0-2-v1_reward/vocab.json
shuttleai-shuttle-2-5-e0-2-v1-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/shuttleai-shuttle-2-5-e0-2-v1_reward/tokenizer.json
Job shuttleai-shuttle-2-5-e0-2-v1-mkmlizer completed after 477.04s with status: succeeded
Stopping job with name shuttleai-shuttle-2-5-e0-2-v1-mkmlizer
Pipeline stage MKMLizer completed in 478.15s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service shuttleai-shuttle-2-5-e0-2-v1
Waiting for inference service shuttleai-shuttle-2-5-e0-2-v1 to be ready
Inference service shuttleai-shuttle-2-5-e0-2-v1 ready after 120.77274250984192s
Pipeline stage ISVCDeployer completed in 122.59s
Running pipeline stage StressChecker
%s, retrying in %s seconds...
%s, retrying in %s seconds...
DeploymentChecksError("expected string or bytes-like object, got 'ReadTimeout'")
shuttleai-shuttle-2-5-e0-2_v1 status is now failed due to DeploymentManager action
admin requested tearing down of shuttleai-shuttle-2-5-e0-2_v1
Running pipeline stage ISVCDeleter
Checking if service shuttleai-shuttle-2-5-e0-2-v1 is running
Tearing down inference service shuttleai-shuttle-2-5-e0-2-v1
Toredown service shuttleai-shuttle-2-5-e0-2-v1
Pipeline stage ISVCDeleter completed in 3.34s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.1.safetensors from bucket guanaco-mkml-models
Deleting key shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.2.safetensors from bucket guanaco-mkml-models
Deleting key shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.3.safetensors from bucket guanaco-mkml-models
Deleting key shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.4.safetensors from bucket guanaco-mkml-models
Deleting key shuttleai-shuttle-2-5-e0-2-v1/flywheel_model.5.safetensors from bucket guanaco-mkml-models
Deleting key shuttleai-shuttle-2-5-e0-2-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key shuttleai-shuttle-2-5-e0-2-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key shuttleai-shuttle-2-5-e0-2-v1/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Pipeline stage MKMLModelDeleter completed in 4.92s
shuttleai-shuttle-2-5-e0-2_v1 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics