rirv938-llama-8b-big-ret_4237

submission_id: rirv938-llama-8b-big-ret_4237_v1
developer_uid: robert_irvine
status: torndown
model_repo: rirv938/llama_8b_big_retune_6m_732
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
timestamp: 2024-09-25T17:32:12+00:00
model_name: rirv938-llama-8b-big-ret_4237_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-big-ret-4237-v1-mkmlizer
Waiting for job on rirv938-llama-8b-big-ret-4237-v1-mkmlizer to finish
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║     _____            __           __                                ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║       /___/                                                         ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: Downloaded to shared memory in 35.782s
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmp7ge1u0vo, device:0
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: quantized model in 83.973s
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: Processed model rirv938/llama_8b_big_retune_6m_732 in 119.756s
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v1
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v1/config.json
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v1/special_tokens_map.json
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v1/tokenizer_config.json
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v1/tokenizer.json
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v1/flywheel_model.0.safetensors
rirv938-llama-8b-big-ret-4237-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:56,  5.13it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:31,  3.15it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:01,  2.36it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:02,  4.51it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:02,  4.55it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:55,  5.10it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:05,  4.28it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:26,  3.22it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:48,  2.54it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:02,  4.38it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:59,  4.61it/s]
Loading 0:   7%|▋         | 21/291 [00:05<00:59,  4.52it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:16,  3.51it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:36,  2.78it/s]
Loading 0:   9%|▉         | 26/291 [00:06<00:59,  4.46it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:56,  4.68it/s]
Loading 0:  10%|█         | 30/291 [00:07<00:56,  4.58it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:12,  3.57it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:31,  2.84it/s]
Loading 0:  12%|█▏        | 35/291 [00:08<00:56,  4.50it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:54,  4.71it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:54,  4.60it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:09,  3.59it/s]
Loading 0:  14%|█▍        | 41/291 [00:10<01:27,  2.85it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:54,  4.57it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:51,  4.77it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<00:52,  4.62it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:07,  3.60it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:23,  2.87it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:51,  4.59it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:49,  4.80it/s]
Loading 0:  19%|█▉        | 55/291 [00:13<00:44,  5.32it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:52,  4.44it/s]
Loading 0:  20%|█▉        | 58/291 [00:14<01:09,  3.36it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:26,  2.67it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:50,  4.51it/s]
Loading 0:  22%|██▏       | 63/291 [00:15<00:48,  4.74it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:49,  4.58it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:03,  3.55it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:19,  2.81it/s]
Loading 0:  24%|██▍       | 71/291 [00:17<00:48,  4.50it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:45,  4.79it/s]
Loading 0:  26%|██▌       | 75/291 [00:18<00:46,  4.64it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<00:59,  3.62it/s]
Loading 0:  26%|██▋       | 77/291 [00:19<01:14,  2.88it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:46,  4.56it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:43,  4.78it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:37,  5.50it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:53,  3.88it/s]
Loading 0:  29%|██▉       | 85/291 [00:21<01:07,  3.07it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:22,  2.49it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:46,  4.35it/s]
Loading 0:  31%|███       | 90/291 [00:22<00:43,  4.60it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:43,  4.54it/s]
Loading 0:  32%|███▏      | 94/291 [00:23<00:55,  3.52it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:09,  2.82it/s]
Loading 0:  34%|███▎      | 98/291 [00:24<00:42,  4.56it/s]
Loading 0:  34%|███▍      | 99/291 [00:24<00:40,  4.77it/s]
Loading 0:  34%|███▍      | 100/291 [00:25<00:36,  5.30it/s]
Loading 0:  35%|███▌      | 102/291 [00:25<00:42,  4.44it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:56,  3.35it/s]
Loading 0:  36%|███▌      | 104/291 [00:26<01:10,  2.65it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:40,  4.51it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:38,  4.75it/s]
Loading 0:  38%|███▊      | 111/291 [00:27<00:39,  4.61it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:50,  3.57it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:02,  2.83it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:38,  4.51it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:36,  4.73it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:37,  4.62it/s]
Loading 0:  42%|████▏     | 121/291 [00:30<00:47,  3.60it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<00:58,  2.87it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:36,  4.59it/s]
Loading 0:  43%|████▎     | 126/291 [00:31<00:34,  4.80it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:34,  4.64it/s]
Loading 0:  45%|████▍     | 130/291 [00:32<00:44,  3.61it/s]
Loading 0:  45%|████▌     | 131/291 [00:33<00:55,  2.88it/s]
Loading 0:  46%|████▌     | 134/291 [00:33<00:34,  4.60it/s]
Loading 0:  46%|████▋     | 135/291 [00:33<00:32,  4.81it/s]
Loading 0:  47%|████▋     | 138/291 [00:34<00:32,  4.66it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:41,  3.62it/s]
Loading 0:  48%|████▊     | 140/291 [00:35<00:52,  2.86it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:32,  4.58it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:30,  4.79it/s]
Loading 0:  51%|█████     | 147/291 [00:36<00:30,  4.66it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:39,  3.62it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:49,  2.89it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:30,  4.61it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:28,  4.82it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:28,  4.68it/s]
Loading 0:  54%|█████▍    | 157/291 [00:39<00:36,  3.64it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:46,  2.89it/s]
Loading 0:  55%|█████▌    | 161/291 [00:40<00:28,  4.63it/s]
Loading 0:  56%|█████▌    | 162/291 [00:40<00:26,  4.83it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:26,  4.69it/s]
Loading 0:  57%|█████▋    | 166/291 [00:41<00:34,  3.65it/s]
Loading 0:  57%|█████▋    | 167/291 [00:42<00:42,  2.89it/s]
Loading 0:  58%|█████▊    | 170/291 [00:42<00:26,  4.64it/s]
Loading 0:  59%|█████▉    | 171/291 [00:42<00:24,  4.84it/s]
Loading 0:  59%|█████▉    | 173/291 [00:43<00:29,  4.02it/s]
Loading 0:  60%|██████    | 175/291 [00:43<00:23,  4.99it/s]
Loading 0:  60%|██████    | 176/291 [00:43<00:22,  5.17it/s]
Loading 0:  62%|██████▏   | 179/291 [00:44<00:23,  4.86it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:29,  3.71it/s]
Loading 0:  62%|██████▏   | 181/291 [00:45<00:37,  2.93it/s]
Loading 0:  63%|██████▎   | 184/291 [00:45<00:22,  4.71it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:21,  4.92it/s]
Loading 0:  64%|██████▍   | 187/291 [00:46<00:18,  5.65it/s]
Loading 0:  65%|██████▍   | 188/291 [00:46<00:26,  3.95it/s]
Loading 0:  65%|██████▍   | 189/291 [00:47<00:34,  2.99it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:26,  3.78it/s]
Loading 0:  66%|██████▋   | 193/291 [00:48<00:31,  3.11it/s]
Loading 0:  67%|██████▋   | 194/291 [00:49<00:37,  2.57it/s]
Loading 0:  68%|██████▊   | 197/291 [00:49<00:21,  4.28it/s]
Loading 0:  68%|██████▊   | 198/291 [00:49<00:20,  4.52it/s]
Loading 0:  69%|██████▉   | 201/291 [00:50<00:19,  4.53it/s]
Loading 0:  69%|██████▉   | 202/291 [00:50<00:25,  3.56it/s]
Loading 0:  70%|██████▉   | 203/291 [00:51<00:30,  2.86it/s]
Loading 0:  71%|███████   | 206/291 [00:51<00:18,  4.55it/s]
Loading 0:  71%|███████   | 207/291 [00:51<00:17,  4.77it/s]
Loading 0:  72%|███████▏  | 210/291 [00:52<00:17,  4.66it/s]
Loading 0:  73%|███████▎  | 211/291 [00:53<00:22,  3.63it/s]
Loading 0:  73%|███████▎  | 212/291 [00:53<00:27,  2.89it/s]
Loading 0:  74%|███████▍  | 215/291 [00:53<00:16,  4.62it/s]
Loading 0:  74%|███████▍  | 216/291 [00:54<00:15,  4.83it/s]
Loading 0:  75%|███████▌  | 219/291 [00:54<00:15,  4.67it/s]
Loading 0:  76%|███████▌  | 220/291 [00:55<00:19,  3.64it/s]
Loading 0:  76%|███████▌  | 221/291 [00:55<00:24,  2.91it/s]
Loading 0:  77%|███████▋  | 224/291 [00:56<00:14,  4.64it/s]
Loading 0:  77%|███████▋  | 225/291 [00:56<00:13,  4.85it/s]
Loading 0:  78%|███████▊  | 226/291 [00:56<00:12,  5.41it/s]
Loading 0:  78%|███████▊  | 228/291 [00:57<00:13,  4.50it/s]
Loading 0:  79%|███████▊  | 229/291 [00:57<00:18,  3.40it/s]
Loading 0:  79%|███████▉  | 230/291 [00:58<00:22,  2.69it/s]
Loading 0:  80%|████████  | 233/291 [00:58<00:12,  4.57it/s]
Loading 0:  80%|████████  | 234/291 [00:58<00:11,  4.80it/s]
Loading 0:  81%|████████▏ | 237/291 [00:59<00:11,  4.66it/s]
Loading 0:  82%|████████▏ | 238/291 [00:59<00:14,  3.62it/s]
Loading 0:  82%|████████▏ | 239/291 [01:00<00:18,  2.89it/s]
Loading 0:  83%|████████▎ | 242/291 [01:00<00:10,  4.66it/s]
Loading 0:  84%|████████▎ | 243/291 [01:00<00:09,  4.87it/s]
Loading 0:  85%|████████▍ | 246/291 [01:01<00:09,  4.68it/s]
Loading 0:  85%|████████▍ | 247/291 [01:02<00:12,  3.65it/s]
Loading 0:  85%|████████▌ | 248/291 [01:02<00:14,  2.91it/s]
Loading 0:  86%|████████▋ | 251/291 [01:02<00:08,  4.60it/s]
Loading 0:  87%|████████▋ | 252/291 [01:03<00:08,  4.81it/s]
Loading 0:  88%|████████▊ | 255/291 [01:03<00:07,  4.67it/s]
Loading 0:  88%|████████▊ | 256/291 [01:04<00:09,  3.65it/s]
Loading 0:  88%|████████▊ | 257/291 [01:04<00:11,  2.91it/s]
Loading 0:  89%|████████▉ | 260/291 [01:05<00:06,  4.66it/s]
Loading 0:  90%|████████▉ | 261/291 [01:05<00:06,  4.86it/s]
Loading 0:  91%|█████████ | 264/291 [01:05<00:05,  4.69it/s]
Loading 0:  91%|█████████ | 265/291 [01:06<00:07,  3.66it/s]
Loading 0:  91%|█████████▏| 266/291 [01:07<00:08,  2.92it/s]
Loading 0:  92%|█████████▏| 269/291 [01:07<00:04,  4.67it/s]
Loading 0:  93%|█████████▎| 270/291 [01:07<00:04,  4.88it/s]
Loading 0:  94%|█████████▍| 273/291 [01:08<00:03,  4.72it/s]
Loading 0:  94%|█████████▍| 274/291 [01:08<00:04,  3.67it/s]
Loading 0:  95%|█████████▍| 275/291 [01:09<00:05,  2.92it/s]
Loading 0:  96%|█████████▌| 278/291 [01:09<00:02,  4.64it/s]
Loading 0:  96%|█████████▌| 279/291 [01:09<00:02,  4.85it/s]
Loading 0:  96%|█████████▌| 280/291 [01:09<00:02,  5.36it/s]
Loading 0:  97%|█████████▋| 281/291 [01:10<00:02,  3.67it/s]
Loading 0:  97%|█████████▋| 282/291 [01:10<00:03,  2.79it/s]
Loading 0:  98%|█████████▊| 284/291 [01:11<00:01,  4.00it/s]
Loading 0:  98%|█████████▊| 285/291 [01:11<00:01,  4.34it/s]
Loading 0:  99%|█████████▊| 287/291 [01:11<00:00,  5.37it/s]
Loading 0:  99%|█████████▉| 288/291 [01:12<00:00,  3.70it/s]
                                                            
Connection pool is full, discarding connection: %s. Connection pool size: %s
Job rirv938-llama-8b-big-ret-4237-v1-mkmlizer completed after 194.69s with status: succeeded
Stopping job with name rirv938-llama-8b-big-ret-4237-v1-mkmlizer
Pipeline stage MKMLizer completed in 195.53s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-big-ret-4237-v1
Waiting for inference service rirv938-llama-8b-big-ret-4237-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service rirv938-llama-8b-big-ret-4237-v1 ready after 221.406498670578s
Pipeline stage MKMLDeployer completed in 222.25s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 6.236687183380127s
Received healthy response to inference request in 5.843423366546631s
Received healthy response to inference request in 5.908728361129761s
Received healthy response to inference request in 4.0496532917022705s
Received healthy response to inference request in 3.946617603302002s
5 requests
0 failed requests
5th percentile: 3.9672247409820556
10th percentile: 3.987831878662109
20th percentile: 4.029046154022216
30th percentile: 4.408407306671142
40th percentile: 5.1259153366088865
50th percentile: 5.843423366546631
60th percentile: 5.869545364379883
70th percentile: 5.895667362213135
80th percentile: 5.974320125579834
90th percentile: 6.10550365447998
95th percentile: 6.1710954189300535
99th percentile: 6.2235688304901124
mean time: 5.197021961212158
%s, retrying in %s seconds...
Received healthy response to inference request in 5.145348787307739s
Received healthy response to inference request in 5.56738018989563s
Received healthy response to inference request in 5.786538124084473s
Received healthy response to inference request in 5.2072858810424805s
Received healthy response to inference request in 5.846510410308838s
5 requests
0 failed requests
5th percentile: 5.157736206054688
10th percentile: 5.170123624801636
20th percentile: 5.194898462295532
30th percentile: 5.27930474281311
40th percentile: 5.42334246635437
50th percentile: 5.56738018989563
60th percentile: 5.655043363571167
70th percentile: 5.742706537246704
80th percentile: 5.798532581329345
90th percentile: 5.822521495819092
95th percentile: 5.834515953063965
99th percentile: 5.844111518859863
mean time: 5.510612678527832
%s, retrying in %s seconds...
Received healthy response to inference request in 3.346173048019409s
Received healthy response to inference request in 6.0028815269470215s
Received healthy response to inference request in 4.545010328292847s
Received healthy response to inference request in 4.853931188583374s
Received healthy response to inference request in 4.902661085128784s
5 requests
0 failed requests
5th percentile: 3.5859405040740966
10th percentile: 3.825707960128784
20th percentile: 4.305242872238159
30th percentile: 4.6067945003509525
40th percentile: 4.730362844467163
50th percentile: 4.853931188583374
60th percentile: 4.873423147201538
70th percentile: 4.892915105819702
80th percentile: 5.122705173492432
90th percentile: 5.562793350219726
95th percentile: 5.782837438583374
99th percentile: 5.958872709274292
mean time: 4.730131435394287
clean up pipeline due to error=%s
Shutdown handler de-registered
rirv938-llama-8b-big-ret_4237_v1 status is now failed due to DeploymentManager action
rirv938-llama-8b-big-ret_4237_v1 status is now torndown due to DeploymentManager action