rirv938-llama-8b-big-ret_3238

submission_id: rirv938-llama-8b-big-ret_3238_v1
developer_uid: robert_irvine
status: torndown
model_repo: rirv938/llama_8b_big_retune_6m_8052
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
timestamp: 2024-09-25T17:34:14+00:00
model_name: rirv938-llama-8b-big-ret_3238_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-big-ret-3238-v1-mkmlizer
Waiting for job on rirv938-llama-8b-big-ret-3238-v1-mkmlizer to finish
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║     _____            __           __                                ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║       /___/                                                         ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: Downloaded to shared memory in 34.852s
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpdcrqcval, device:0
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: quantized model in 85.313s
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: Processed model rirv938/llama_8b_big_retune_6m_8052 in 120.166s
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-3238-v1
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-3238-v1/config.json
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-3238-v1/special_tokens_map.json
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-3238-v1/tokenizer_config.json
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-3238-v1/tokenizer.json
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-3238-v1/flywheel_model.0.safetensors
rirv938-llama-8b-big-ret-3238-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:57,  5.05it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:32,  3.11it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:02,  2.33it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:03,  4.45it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:02,  4.51it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:54,  5.14it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:04,  4.29it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:26,  3.23it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:48,  2.56it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:02,  4.39it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:59,  4.62it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:52,  5.19it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:02,  4.33it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:22,  3.26it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:44,  2.57it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:00,  4.38it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:57,  4.59it/s]
Loading 0:  10%|▉         | 28/291 [00:06<00:51,  5.12it/s]
Loading 0:  10%|█         | 30/291 [00:07<01:00,  4.33it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:19,  3.27it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:39,  2.59it/s]
Loading 0:  12%|█▏        | 35/291 [00:08<00:58,  4.41it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:54,  4.65it/s]
Loading 0:  13%|█▎        | 37/291 [00:09<00:48,  5.25it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:57,  4.37it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:15,  3.30it/s]
Loading 0:  14%|█▍        | 41/291 [00:11<01:35,  2.61it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:55,  4.43it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:52,  4.67it/s]
Loading 0:  16%|█▌        | 46/291 [00:11<00:47,  5.19it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<00:55,  4.36it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:13,  3.30it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:32,  2.61it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:53,  4.43it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:50,  4.66it/s]
Loading 0:  19%|█▉        | 55/291 [00:13<00:45,  5.19it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:53,  4.35it/s]
Loading 0:  20%|█▉        | 58/291 [00:14<01:10,  3.28it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:29,  2.59it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:51,  4.45it/s]
Loading 0:  22%|██▏       | 63/291 [00:16<00:48,  4.69it/s]
Loading 0:  22%|██▏       | 64/291 [00:16<00:43,  5.27it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:51,  4.40it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:07,  3.31it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:26,  2.59it/s]
Loading 0:  24%|██▍       | 71/291 [00:18<00:49,  4.41it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:46,  4.73it/s]
Loading 0:  25%|██▌       | 73/291 [00:18<00:41,  5.30it/s]
Loading 0:  26%|██▌       | 75/291 [00:18<00:49,  4.40it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<01:04,  3.32it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<01:22,  2.61it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:47,  4.43it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:45,  4.67it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:39,  5.26it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:39,  5.28it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:59,  3.48it/s]
Loading 0:  29%|██▉       | 85/291 [00:22<01:14,  2.75it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:30,  2.27it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:48,  4.19it/s]
Loading 0:  31%|███       | 90/291 [00:23<00:45,  4.46it/s]
Loading 0:  31%|███▏      | 91/291 [00:23<00:39,  5.08it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:46,  4.28it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<01:00,  3.24it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:16,  2.56it/s]
Loading 0:  34%|███▎      | 98/291 [00:25<00:43,  4.40it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:41,  4.64it/s]
Loading 0:  34%|███▍      | 100/291 [00:25<00:36,  5.25it/s]
Loading 0:  35%|███▌      | 102/291 [00:25<00:43,  4.38it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:56,  3.31it/s]
Loading 0:  36%|███▌      | 104/291 [00:27<01:12,  2.59it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:41,  4.45it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:39,  4.69it/s]
Loading 0:  37%|███▋      | 109/291 [00:27<00:34,  5.28it/s]
Loading 0:  38%|███▊      | 111/291 [00:28<00:40,  4.39it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:54,  3.31it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:08,  2.62it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:38,  4.49it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:36,  4.73it/s]
Loading 0:  41%|████      | 118/291 [00:29<00:32,  5.33it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:38,  4.41it/s]
Loading 0:  42%|████▏     | 121/291 [00:31<00:51,  3.31it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<01:04,  2.61it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:37,  4.47it/s]
Loading 0:  43%|████▎     | 126/291 [00:32<00:35,  4.71it/s]
Loading 0:  44%|████▎     | 127/291 [00:32<00:30,  5.31it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:36,  4.42it/s]
Loading 0:  45%|████▍     | 130/291 [00:33<00:48,  3.32it/s]
Loading 0:  45%|████▌     | 131/291 [00:33<01:01,  2.62it/s]
Loading 0:  46%|████▌     | 134/291 [00:34<00:35,  4.48it/s]
Loading 0:  46%|████▋     | 135/291 [00:34<00:33,  4.72it/s]
Loading 0:  47%|████▋     | 137/291 [00:34<00:23,  6.47it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:45,  3.36it/s]
Loading 0:  48%|████▊     | 140/291 [00:36<00:55,  2.74it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:33,  4.39it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:31,  4.61it/s]
Loading 0:  50%|████▉     | 145/291 [00:36<00:28,  5.11it/s]
Loading 0:  51%|█████     | 147/291 [00:37<00:33,  4.35it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:43,  3.32it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:53,  2.64it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:31,  4.44it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:29,  4.67it/s]
Loading 0:  53%|█████▎    | 154/291 [00:38<00:26,  5.24it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:30,  4.40it/s]
Loading 0:  54%|█████▍    | 157/291 [00:40<00:40,  3.32it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:50,  2.62it/s]
Loading 0:  55%|█████▌    | 161/291 [00:40<00:28,  4.50it/s]
Loading 0:  56%|█████▌    | 162/291 [00:41<00:27,  4.73it/s]
Loading 0:  56%|█████▌    | 163/291 [00:41<00:24,  5.32it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:28,  4.42it/s]
Loading 0:  57%|█████▋    | 166/291 [00:42<00:37,  3.33it/s]
Loading 0:  57%|█████▋    | 167/291 [00:43<00:47,  2.63it/s]
Loading 0:  58%|█████▊    | 170/291 [00:43<00:27,  4.47it/s]
Loading 0:  59%|█████▉    | 171/291 [00:43<00:25,  4.70it/s]
Loading 0:  59%|█████▉    | 172/291 [00:43<00:22,  5.30it/s]
Loading 0:  59%|█████▉    | 173/291 [00:44<00:32,  3.58it/s]
Loading 0:  60%|██████    | 175/291 [00:44<00:23,  4.90it/s]
Loading 0:  60%|██████    | 176/291 [00:44<00:22,  5.12it/s]
Loading 0:  61%|██████    | 177/291 [00:44<00:19,  5.80it/s]
Loading 0:  62%|██████▏   | 179/291 [00:45<00:24,  4.55it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:33,  3.33it/s]
Loading 0:  62%|██████▏   | 181/291 [00:46<00:41,  2.62it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:23,  4.59it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:21,  4.83it/s]
Loading 0:  64%|██████▍   | 186/291 [00:46<00:19,  5.44it/s]
Loading 0:  64%|██████▍   | 187/291 [00:46<00:19,  5.45it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:29,  3.54it/s]
Loading 0:  65%|██████▍   | 189/291 [00:48<00:38,  2.64it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:27,  3.58it/s]
Loading 0:  66%|██████▋   | 193/291 [00:49<00:33,  2.95it/s]
Loading 0:  67%|██████▋   | 194/291 [00:49<00:39,  2.47it/s]
Loading 0:  68%|██████▊   | 197/291 [00:50<00:22,  4.16it/s]
Loading 0:  68%|██████▊   | 198/291 [00:50<00:21,  4.43it/s]
Loading 0:  69%|██████▉   | 201/291 [00:50<00:20,  4.45it/s]
Loading 0:  69%|██████▉   | 202/291 [00:51<00:25,  3.48it/s]
Loading 0:  70%|██████▉   | 203/291 [00:52<00:31,  2.80it/s]
Loading 0:  71%|███████   | 206/291 [00:52<00:18,  4.47it/s]
Loading 0:  71%|███████   | 207/291 [00:52<00:17,  4.69it/s]
Loading 0:  71%|███████▏  | 208/291 [00:52<00:15,  5.25it/s]
Loading 0:  72%|███████▏  | 210/291 [00:53<00:18,  4.43it/s]
Loading 0:  73%|███████▎  | 211/291 [00:53<00:23,  3.35it/s]
Loading 0:  73%|███████▎  | 212/291 [00:54<00:29,  2.66it/s]
Loading 0:  74%|███████▍  | 215/291 [00:54<00:16,  4.48it/s]
Loading 0:  74%|███████▍  | 216/291 [00:54<00:15,  4.71it/s]
Loading 0:  75%|███████▍  | 217/291 [00:54<00:13,  5.30it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:16,  4.41it/s]
Loading 0:  76%|███████▌  | 220/291 [00:56<00:21,  3.33it/s]
Loading 0:  76%|███████▌  | 221/291 [00:56<00:26,  2.63it/s]
Loading 0:  77%|███████▋  | 224/291 [00:56<00:14,  4.51it/s]
Loading 0:  77%|███████▋  | 225/291 [00:57<00:13,  4.74it/s]
Loading 0:  78%|███████▊  | 226/291 [00:57<00:12,  5.34it/s]
Loading 0:  78%|███████▊  | 228/291 [00:57<00:14,  4.42it/s]
Loading 0:  79%|███████▊  | 229/291 [00:58<00:18,  3.33it/s]
Loading 0:  79%|███████▉  | 230/291 [00:58<00:23,  2.64it/s]
Loading 0:  80%|████████  | 233/291 [00:59<00:12,  4.52it/s]
Loading 0:  80%|████████  | 234/291 [00:59<00:11,  4.75it/s]
Loading 0:  81%|████████  | 235/291 [00:59<00:10,  5.30it/s]
Loading 0:  81%|████████▏ | 237/291 [01:00<00:12,  4.41it/s]
Loading 0:  82%|████████▏ | 238/291 [01:00<00:15,  3.33it/s]
Loading 0:  82%|████████▏ | 239/291 [01:01<00:19,  2.65it/s]
Loading 0:  83%|████████▎ | 242/291 [01:01<00:10,  4.55it/s]
Loading 0:  84%|████████▎ | 243/291 [01:01<00:10,  4.78it/s]
Loading 0:  84%|████████▍ | 244/291 [01:01<00:08,  5.38it/s]
Loading 0:  85%|████████▍ | 246/291 [01:02<00:10,  4.44it/s]
Loading 0:  85%|████████▍ | 247/291 [01:02<00:13,  3.35it/s]
Loading 0:  85%|████████▌ | 248/291 [01:03<00:16,  2.65it/s]
Loading 0:  86%|████████▋ | 251/291 [01:03<00:08,  4.55it/s]
Loading 0:  87%|████████▋ | 252/291 [01:03<00:08,  4.77it/s]
Loading 0:  87%|████████▋ | 253/291 [01:03<00:07,  5.36it/s]
Loading 0:  88%|████████▊ | 255/291 [01:04<00:08,  4.43it/s]
Loading 0:  88%|████████▊ | 256/291 [01:05<00:10,  3.34it/s]
Loading 0:  88%|████████▊ | 257/291 [01:05<00:12,  2.64it/s]
Loading 0:  89%|████████▉ | 260/291 [01:05<00:06,  4.54it/s]
Loading 0:  90%|████████▉ | 261/291 [01:06<00:06,  4.77it/s]
Loading 0:  90%|█████████ | 262/291 [01:06<00:05,  5.36it/s]
Loading 0:  91%|█████████ | 264/291 [01:06<00:06,  4.43it/s]
Loading 0:  91%|█████████ | 265/291 [01:07<00:07,  3.34it/s]
Loading 0:  91%|█████████▏| 266/291 [01:07<00:09,  2.65it/s]
Loading 0:  92%|█████████▏| 269/291 [01:08<00:04,  4.55it/s]
Loading 0:  93%|█████████▎| 270/291 [01:08<00:04,  4.76it/s]
Loading 0:  93%|█████████▎| 271/291 [01:08<00:03,  5.35it/s]
Loading 0:  94%|█████████▍| 273/291 [01:08<00:04,  4.44it/s]
Loading 0:  94%|█████████▍| 274/291 [01:09<00:05,  3.33it/s]
Loading 0:  95%|█████████▍| 275/291 [01:10<00:06,  2.62it/s]
Loading 0:  96%|█████████▌| 278/291 [01:10<00:02,  4.50it/s]
Loading 0:  96%|█████████▌| 279/291 [01:10<00:02,  4.74it/s]
Loading 0:  96%|█████████▌| 280/291 [01:10<00:02,  5.34it/s]
Loading 0:  97%|█████████▋| 281/291 [01:11<00:02,  3.59it/s]
Loading 0:  97%|█████████▋| 282/291 [01:11<00:03,  2.71it/s]
Loading 0:  98%|█████████▊| 284/291 [01:12<00:01,  3.89it/s]
Loading 0:  98%|█████████▊| 285/291 [01:12<00:01,  4.24it/s]
Loading 0:  98%|█████████▊| 286/291 [01:12<00:01,  4.93it/s]
Loading 0:  99%|█████████▊| 287/291 [01:12<00:00,  5.12it/s]
Loading 0:  99%|█████████▉| 288/291 [01:13<00:00,  3.35it/s]
                                                            
Job rirv938-llama-8b-big-ret-3238-v1-mkmlizer completed after 147.17s with status: succeeded
Stopping job with name rirv938-llama-8b-big-ret-3238-v1-mkmlizer
Pipeline stage MKMLizer completed in 147.97s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-big-ret-3238-v1
Waiting for inference service rirv938-llama-8b-big-ret-3238-v1 to be ready
Inference service rirv938-llama-8b-big-ret-3238-v1 ready after 221.3298363685608s
Pipeline stage MKMLDeployer completed in 221.70s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 6.066540956497192s
Received healthy response to inference request in 4.798480272293091s
Received healthy response to inference request in 4.913106441497803s
Received healthy response to inference request in 4.500491380691528s
Received healthy response to inference request in 5.389132738113403s
5 requests
0 failed requests
5th percentile: 4.560089159011841
10th percentile: 4.619686937332153
20th percentile: 4.7388824939727785
30th percentile: 4.821405506134033
40th percentile: 4.867255973815918
50th percentile: 4.913106441497803
60th percentile: 5.103516960144043
70th percentile: 5.293927478790283
80th percentile: 5.524614381790161
90th percentile: 5.795577669143677
95th percentile: 5.931059312820435
99th percentile: 6.039444627761841
mean time: 5.133550357818604
%s, retrying in %s seconds...
Received healthy response to inference request in 4.323946952819824s
Received healthy response to inference request in 5.872666597366333s
Received healthy response to inference request in 5.408345699310303s
Received healthy response to inference request in 5.217276334762573s
Received healthy response to inference request in 5.3362109661102295s
5 requests
0 failed requests
5th percentile: 4.502612829208374
10th percentile: 4.681278705596924
20th percentile: 5.038610458374023
30th percentile: 5.241063261032105
40th percentile: 5.288637113571167
50th percentile: 5.3362109661102295
60th percentile: 5.365064859390259
70th percentile: 5.393918752670288
80th percentile: 5.501209878921509
90th percentile: 5.686938238143921
95th percentile: 5.779802417755127
99th percentile: 5.8540937614440915
mean time: 5.231689310073852
%s, retrying in %s seconds...
Received healthy response to inference request in 4.993000507354736s
Received healthy response to inference request in 6.358447074890137s
Received healthy response to inference request in 5.338910341262817s
Received healthy response to inference request in 5.187068700790405s
Received healthy response to inference request in 6.246040344238281s
5 requests
0 failed requests
5th percentile: 5.03181414604187
10th percentile: 5.0706277847290036
20th percentile: 5.148255062103272
30th percentile: 5.217437028884888
40th percentile: 5.2781736850738525
50th percentile: 5.338910341262817
60th percentile: 5.701762342453003
70th percentile: 6.064614343643188
80th percentile: 6.268521690368653
90th percentile: 6.313484382629395
95th percentile: 6.335965728759765
99th percentile: 6.353950805664063
mean time: 5.624693393707275
clean up pipeline due to error=%s
Shutdown handler de-registered
rirv938-llama-8b-big-ret_3238_v1 status is now failed due to DeploymentManager action
rirv938-llama-8b-big-ret_3238_v1 status is now torndown due to DeploymentManager action