submission_id: rirv938-llama-8b-big-ret_4805_v3
developer_uid: robert_irvine
status: torndown
model_repo: rirv938/llama_8b_big_retune_6m_4392
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
timestamp: 2024-09-25T18:18:40+00:00
model_name: rirv938-llama-8b-big-ret_4805_v3
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-big-ret-4805-v3-mkmlizer
Waiting for job on rirv938-llama-8b-big-ret-4805-v3-mkmlizer to finish
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ _____ __ __ ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ /___/ ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ Version: 0.11.12 ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ https://mk1.ai ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ belonging to: ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ Chai Research Corp. ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ║ ║
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: Downloaded to shared memory in 21.233s
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmprrp34yhm, device:0
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: quantized model in 83.275s
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: Processed model rirv938/llama_8b_big_retune_6m_4392 in 104.508s
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v3
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v3/config.json
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v3/special_tokens_map.json
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v3/tokenizer_config.json
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v3/flywheel_model.0.safetensors
rirv938-llama-8b-big-ret-4805-v3-mkmlizer: Loading 0: 0%| | 0/291 [00:00<?, ?it/s] Loading 0: 1%| | 3/291 [00:00<00:54, 5.24it/s] Loading 0: 1%|▏ | 4/291 [00:01<01:29, 3.20it/s] Loading 0: 2%|▏ | 5/291 [00:01<01:59, 2.40it/s] Loading 0: 3%|▎ | 8/291 [00:02<01:01, 4.57it/s] Loading 0: 3%|▎ | 9/291 [00:02<01:01, 4.62it/s] Loading 0: 4%|▍ | 12/291 [00:02<01:01, 4.57it/s] Loading 0: 4%|▍ | 13/291 [00:03<01:18, 3.53it/s] Loading 0: 5%|▍ | 14/291 [00:04<01:38, 2.82it/s] Loading 0: 6%|▌ | 17/291 [00:04<01:00, 4.55it/s] Loading 0: 6%|▌ | 18/291 [00:04<00:57, 4.77it/s] Loading 0: 7%|▋ | 19/291 [00:04<00:51, 5.29it/s] Loading 0: 7%|▋ | 21/291 [00:05<01:01, 4.41it/s] Loading 0: 8%|▊ | 22/291 [00:05<01:20, 3.35it/s] Loading 0: 8%|▊ | 23/291 [00:06<01:41, 2.65it/s] Loading 0: 9%|▉ | 26/291 [00:06<00:59, 4.48it/s] Loading 0: 9%|▉ | 27/291 [00:06<00:55, 4.72it/s] Loading 0: 10%|█ | 30/291 [00:07<00:56, 4.64it/s] Loading 0: 11%|█ | 31/291 [00:07<01:12, 3.61it/s] Loading 0: 11%|█ | 32/291 [00:08<01:30, 2.87it/s] Loading 0: 12%|█▏ | 35/291 [00:08<00:55, 4.59it/s] Loading 0: 12%|█▏ | 36/291 [00:08<00:53, 4.81it/s] Loading 0: 13%|█▎ | 37/291 [00:09<00:47, 5.37it/s] Loading 0: 13%|█▎ | 39/291 [00:09<00:56, 4.50it/s] Loading 0: 14%|█▎ | 40/291 [00:10<01:13, 3.39it/s] Loading 0: 14%|█▍ | 41/291 [00:10<01:32, 2.69it/s] Loading 0: 15%|█▌ | 44/291 [00:10<00:54, 4.54it/s] Loading 0: 15%|█▌ | 45/291 [00:11<00:51, 4.78it/s] Loading 0: 16%|█▋ | 48/291 [00:11<00:51, 4.68it/s] Loading 0: 17%|█▋ | 49/291 [00:12<01:06, 3.63it/s] Loading 0: 17%|█▋ | 50/291 [00:12<01:23, 2.90it/s] Loading 0: 18%|█▊ | 53/291 [00:13<00:51, 4.59it/s] Loading 0: 19%|█▊ | 54/291 [00:13<00:49, 4.82it/s] Loading 0: 20%|█▉ | 57/291 [00:14<00:49, 4.69it/s] Loading 0: 20%|█▉ | 58/291 [00:14<01:03, 3.66it/s] Loading 0: 20%|██ | 59/291 [00:15<01:20, 2.89it/s] Loading 0: 21%|██▏ | 62/291 [00:15<00:50, 4.56it/s] Loading 0: 22%|██▏ | 63/291 [00:15<00:47, 4.79it/s] Loading 0: 22%|██▏ | 64/291 [00:15<00:42, 5.35it/s] Loading 0: 23%|██▎ | 66/291 [00:16<00:50, 4.49it/s] Loading 0: 23%|██▎ | 67/291 [00:16<01:05, 3.42it/s] Loading 0: 23%|██▎ | 68/291 [00:17<01:22, 2.70it/s] Loading 0: 24%|██▍ | 71/291 [00:17<00:48, 4.52it/s] Loading 0: 25%|██▍ | 72/291 [00:17<00:45, 4.76it/s] Loading 0: 26%|██▌ | 75/291 [00:18<00:46, 4.65it/s] Loading 0: 26%|██▌ | 76/291 [00:19<00:59, 3.62it/s] Loading 0: 26%|██▋ | 77/291 [00:19<01:14, 2.88it/s] Loading 0: 27%|██▋ | 80/291 [00:19<00:46, 4.58it/s] Loading 0: 28%|██▊ | 81/291 [00:20<00:43, 4.80it/s] Loading 0: 29%|██▊ | 83/291 [00:20<00:37, 5.49it/s] Loading 0: 29%|██▉ | 84/291 [00:20<00:53, 3.89it/s] Loading 0: 29%|██▉ | 85/291 [00:21<01:07, 3.06it/s] Loading 0: 30%|██▉ | 86/291 [00:22<01:21, 2.52it/s] Loading 0: 31%|███ | 89/291 [00:22<00:46, 4.34it/s] Loading 0: 31%|███ | 90/291 [00:22<00:43, 4.60it/s] Loading 0: 32%|███▏ | 93/291 [00:23<00:43, 4.56it/s] Loading 0: 32%|███▏ | 94/291 [00:23<00:55, 3.56it/s] Loading 0: 33%|███▎ | 95/291 [00:24<01:08, 2.84it/s] Loading 0: 34%|███▎ | 98/291 [00:24<00:42, 4.54it/s] Loading 0: 34%|███▍ | 99/291 [00:24<00:40, 4.76it/s] Loading 0: 35%|███▌ | 102/291 [00:25<00:40, 4.65it/s] Loading 0: 35%|███▌ | 103/291 [00:25<00:51, 3.63it/s] Loading 0: 36%|███▌ | 104/291 [00:26<01:04, 2.89it/s] Loading 0: 37%|███▋ | 107/291 [00:26<00:39, 4.63it/s] Loading 0: 37%|███▋ | 108/291 [00:26<00:37, 4.84it/s] Loading 0: 38%|███▊ | 111/291 [00:27<00:38, 4.70it/s] Loading 0: 38%|███▊ | 112/291 [00:28<00:48, 3.66it/s] Loading 0: 39%|███▉ | 113/291 [00:28<01:01, 2.91it/s] Loading 0: 40%|███▉ | 116/291 [00:28<00:37, 4.64it/s] Loading 0: 40%|████ | 117/291 [00:29<00:35, 4.85it/s] Loading 0: 41%|████ | 120/291 [00:29<00:36, 4.72it/s] Loading 0: 42%|████▏ | 121/291 [00:30<00:46, 3.68it/s] Loading 0: 42%|████▏ | 122/291 [00:30<00:57, 2.93it/s] Loading 0: 43%|████▎ | 125/291 [00:31<00:35, 4.64it/s] Loading 0: 43%|████▎ | 126/291 [00:31<00:33, 4.93it/s] Loading 0: 44%|████▍ | 129/291 [00:31<00:34, 4.75it/s] Loading 0: 45%|████▍ | 130/291 [00:32<00:43, 3.70it/s] Loading 0: 45%|████▌ | 131/291 [00:33<00:54, 2.95it/s] Loading 0: 46%|████▌ | 134/291 [00:33<00:33, 4.71it/s] Loading 0: 46%|████▋ | 135/291 [00:33<00:31, 4.92it/s] Loading 0: 47%|████▋ | 138/291 [00:34<00:32, 4.76it/s] Loading 0: 48%|████▊ | 139/291 [00:34<00:41, 3.70it/s] Loading 0: 48%|████▊ | 140/291 [00:35<00:51, 2.91it/s] Loading 0: 49%|████▉ | 143/291 [00:35<00:31, 4.63it/s] Loading 0: 49%|████▉ | 144/291 [00:35<00:30, 4.84it/s] Loading 0: 50%|█████ | 146/291 [00:35<00:22, 6.51it/s] Loading 0: 51%|█████ | 148/291 [00:36<00:40, 3.50it/s] Loading 0: 51%|█████ | 149/291 [00:37<00:49, 2.87it/s] Loading 0: 52%|█████▏ | 152/291 [00:37<00:30, 4.54it/s] Loading 0: 53%|█████▎ | 153/291 [00:37<00:28, 4.76it/s] Loading 0: 54%|█████▎ | 156/291 [00:38<00:28, 4.68it/s] Loading 0: 54%|█████▍ | 157/291 [00:39<00:36, 3.67it/s] Loading 0: 54%|█████▍ | 158/291 [00:39<00:45, 2.93it/s] Loading 0: 55%|█████▌ | 161/291 [00:39<00:28, 4.62it/s] Loading 0: 56%|█████▌ | 162/291 [00:40<00:26, 4.84it/s] Loading 0: 57%|█████▋ | 165/291 [00:40<00:26, 4.72it/s] Loading 0: 57%|█████▋ | 166/291 [00:41<00:33, 3.69it/s] Loading 0: 57%|█████▋ | 167/291 [00:41<00:42, 2.95it/s] Loading 0: 58%|█████▊ | 170/291 [00:42<00:25, 4.65it/s] Loading 0: 59%|█████▉ | 171/291 [00:42<00:24, 4.87it/s] Loading 0: 59%|█████▉ | 173/291 [00:42<00:29, 4.05it/s] Loading 0: 60%|██████ | 175/291 [00:43<00:22, 5.09it/s] Loading 0: 60%|██████ | 176/291 [00:43<00:21, 5.27it/s] Loading 0: 62%|██████▏ | 179/291 [00:43<00:22, 4.91it/s] Loading 0: 62%|██████▏ | 180/291 [00:44<00:29, 3.76it/s] Loading 0: 62%|██████▏ | 181/291 [00:45<00:37, 2.96it/s] Loading 0: 63%|██████▎ | 184/291 [00:45<00:22, 4.75it/s] Loading 0: 64%|██████▎ | 185/291 [00:45<00:21, 4.95it/s] Loading 0: 64%|██████▍ | 187/291 [00:45<00:18, 5.64it/s] Loading 0: 65%|██████▍ | 188/291 [00:46<00:25, 3.97it/s] Loading 0: 65%|██████▍ | 189/291 [00:46<00:33, 3.02it/s] Loading 0: 66%|██████▌ | 192/291 [00:47<00:25, 3.83it/s] Loading 0: 66%|██████▋ | 193/291 [00:48<00:31, 3.16it/s] Loading 0: 67%|██████▋ | 194/291 [00:48<00:36, 2.63it/s] Loading 0: 68%|██████▊ | 197/291 [00:48<00:21, 4.37it/s] Loading 0: 68%|██████▊ | 198/291 [00:48<00:20, 4.61it/s] Loading 0: 69%|██████▉ | 201/291 [00:49<00:19, 4.61it/s] Loading 0: 69%|██████▉ | 202/291 [00:50<00:24, 3.62it/s] Loading 0: 70%|██████▉ | 203/291 [00:50<00:30, 2.91it/s] Loading 0: 71%|███████ | 206/291 [00:51<00:18, 4.63it/s] Loading 0: 71%|███████ | 207/291 [00:51<00:17, 4.85it/s] Loading 0: 72%|███████▏ | 210/291 [00:51<00:17, 4.72it/s] Loading 0: 73%|███████▎ | 211/291 [00:52<00:21, 3.67it/s] Loading 0: 73%|███████▎ | 212/291 [00:53<00:27, 2.92it/s] Loading 0: 74%|███████▍ | 215/291 [00:53<00:16, 4.67it/s] Loading 0: 74%|███████▍ | 216/291 [00:53<00:15, 4.88it/s] Loading 0: 75%|███████▌ | 219/291 [00:54<00:15, 4.72it/s] Loading 0: 76%|███████▌ | 220/291 [00:54<00:19, 3.69it/s] Loading 0: 76%|███████▌ | 221/291 [00:55<00:23, 2.95it/s] Loading 0: 77%|███████▋ | 224/291 [00:55<00:14, 4.65it/s] Loading 0: 77%|███████▋ | 225/291 [00:55<00:13, 4.86it/s] Loading 0: 78%|███████▊ | 228/291 [00:56<00:13, 4.69it/s] Loading 0: 79%|███████▊ | 229/291 [00:56<00:17, 3.64it/s] Loading 0: 79%|███████▉ | 230/291 [00:57<00:20, 2.91it/s] Loading 0: 80%|████████ | 233/291 [00:57<00:12, 4.57it/s] Loading 0: 80%|████████ | 234/291 [00:57<00:11, 4.77it/s] Loading 0: 81%|████████ | 235/291 [00:57<00:10, 5.32it/s] Loading 0: 81%|████████▏ | 237/291 [00:58<00:12, 4.47it/s] Loading 0: 82%|████████▏ | 238/291 [00:59<00:15, 3.39it/s] Loading 0: 82%|████████▏ | 239/291 [00:59<00:19, 2.69it/s] Loading 0: 83%|████████▎ | 242/291 [00:59<00:10, 4.57it/s] Loading 0: 84%|████████▎ | 243/291 [01:00<00:10, 4.78it/s] Loading 0: 84%|████████▍ | 244/291 [01:00<00:08, 5.38it/s] Loading 0: 85%|████████▍ | 246/291 [01:00<00:10, 4.48it/s] Loading 0: 85%|████████▍ | 247/291 [01:01<00:13, 3.37it/s] Loading 0: 85%|████████▌ | 248/291 [01:01<00:16, 2.68it/s] Loading 0: 86%|████████▋ | 251/291 [01:02<00:08, 4.58it/s] Loading 0: 87%|████████▋ | 252/291 [01:02<00:08, 4.80it/s] Loading 0: 88%|████████▊ | 255/291 [01:02<00:07, 4.66it/s] Loading 0: 88%|████████▊ | 256/291 [01:03<00:09, 3.62it/s] Loading 0: 88%|████████▊ | 257/291 [01:04<00:11, 2.88it/s] Loading 0: 89%|████████▉ | 260/291 [01:04<00:06, 4.64it/s] Loading 0: 90%|████████▉ | 261/291 [01:04<00:06, 4.84it/s] Loading 0: 91%|█████████ | 264/291 [01:05<00:05, 4.68it/s] Loading 0: 91%|█████████ | 265/291 [01:05<00:07, 3.65it/s] Loading 0: 91%|█████████▏| 266/291 [01:06<00:08, 2.89it/s] Loading 0: 92%|█████████▏| 269/291 [01:06<00:04, 4.63it/s] Loading 0: 93%|█████████▎| 270/291 [01:06<00:04, 4.83it/s] Loading 0: 93%|█████████▎| 272/291 [01:06<00:02, 6.49it/s] Loading 0: 94%|█████████▍| 274/291 [01:07<00:04, 3.48it/s] Loading 0: 95%|█████████▍| 275/291 [01:08<00:05, 2.85it/s] Loading 0: 96%|█████████▌| 278/291 [01:08<00:02, 4.45it/s] Loading 0: 96%|█████████▌| 279/291 [01:08<00:02, 4.67it/s] Loading 0: 97%|█████████▋| 281/291 [01:09<00:02, 3.94it/s] Loading 0: 97%|█████████▋| 282/291 [01:10<00:02, 3.04it/s] Loading 0: 98%|█████████▊| 284/291 [01:10<00:01, 4.02it/s] Loading 0: 98%|█████████▊| 285/291 [01:10<00:01, 4.32it/s] Loading 0: 98%|█████████▊| 286/291 [01:10<00:01, 4.94it/s] Loading 0: 99%|█████████▊| 287/291 [01:10<00:00, 5.15it/s] Loading 0: 99%|█████████▉| 288/291 [01:11<00:00, 3.45it/s]
Job rirv938-llama-8b-big-ret-4805-v3-mkmlizer completed after 123.57s with status: succeeded
Stopping job with name rirv938-llama-8b-big-ret-4805-v3-mkmlizer
Pipeline stage MKMLizer completed in 124.48s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-big-ret-4805-v3
Waiting for inference service rirv938-llama-8b-big-ret-4805-v3 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service rirv938-llama-8b-big-ret-4805-v3 ready after 211.55799913406372s
Pipeline stage MKMLDeployer completed in 211.96s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 6.324389934539795s
Received healthy response to inference request in 5.030513048171997s
Received healthy response to inference request in 6.749304294586182s
Received healthy response to inference request in 4.09697413444519s
Received healthy response to inference request in 4.267910003662109s
5 requests
0 failed requests
5th percentile: 4.131161308288574
10th percentile: 4.165348482131958
20th percentile: 4.233722829818726
30th percentile: 4.420430612564087
40th percentile: 4.725471830368042
50th percentile: 5.030513048171997
60th percentile: 5.548063802719116
70th percentile: 6.065614557266235
80th percentile: 6.409372806549072
90th percentile: 6.579338550567627
95th percentile: 6.664321422576904
99th percentile: 6.732307720184326
mean time: 5.2938182830810545
%s, retrying in %s seconds...
Received healthy response to inference request in 3.300992965698242s
Received healthy response to inference request in 3.7974655628204346s
Received healthy response to inference request in 2.823601007461548s
Received healthy response to inference request in 3.5455310344696045s
Received healthy response to inference request in 4.2098352909088135s
5 requests
0 failed requests
5th percentile: 2.919079399108887
10th percentile: 3.0145577907562258
20th percentile: 3.2055145740509032
30th percentile: 3.3499005794525147
40th percentile: 3.4477158069610594
50th percentile: 3.5455310344696045
60th percentile: 3.6463048458099365
70th percentile: 3.7470786571502686
80th percentile: 3.8799395084381105
90th percentile: 4.044887399673462
95th percentile: 4.1273613452911375
99th percentile: 4.193340501785278
mean time: 3.5354851722717284
%s, retrying in %s seconds...
Received healthy response to inference request in 3.26636004447937s
Received healthy response to inference request in 7.265692710876465s
Received healthy response to inference request in 4.470422744750977s
Received healthy response to inference request in 2.245333671569824s
Received healthy response to inference request in 2.8779146671295166s
5 requests
0 failed requests
5th percentile: 2.371849870681763
10th percentile: 2.4983660697937013
20th percentile: 2.751398468017578
30th percentile: 2.955603742599487
40th percentile: 3.110981893539429
50th percentile: 3.26636004447937
60th percentile: 3.7479851245880127
70th percentile: 4.229610204696655
80th percentile: 5.029476737976075
90th percentile: 6.14758472442627
95th percentile: 6.706638717651367
99th percentile: 7.153881912231445
mean time: 4.025144767761231
clean up pipeline due to error=%s
Shutdown handler de-registered
rirv938-llama-8b-big-ret_4805_v3 status is now failed due to DeploymentManager action
rirv938-llama-8b-big-ret_4805_v3 status is now torndown due to DeploymentManager action