rirv938-llama-8b-big-ret_4805

submission_id: rirv938-llama-8b-big-ret_4805_v1
developer_uid: robert_irvine
status: torndown
model_repo: rirv938/llama_8b_big_retune_6m_4392
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
timestamp: 2024-09-25T17:33:10+00:00
model_name: rirv938-llama-8b-big-ret_4805_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-big-ret-4805-v1-mkmlizer
Waiting for job on rirv938-llama-8b-big-ret-4805-v1-mkmlizer to finish
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║     _____            __           __                                ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║       /___/                                                         ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: Downloaded to shared memory in 35.783s
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpad1gi1tr, device:0
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: quantized model in 85.725s
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: Processed model rirv938/llama_8b_big_retune_6m_4392 in 121.508s
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v1
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v1/config.json
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v1/special_tokens_map.json
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v1/tokenizer_config.json
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v1/tokenizer.json
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v1/flywheel_model.0.safetensors
rirv938-llama-8b-big-ret-4805-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:56,  5.10it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:32,  3.12it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:01,  2.36it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:02,  4.49it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:02,  4.51it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:55,  5.07it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:05,  4.25it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:27,  3.19it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:49,  2.54it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:03,  4.33it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:59,  4.57it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:53,  5.07it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:03,  4.28it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:22,  3.24it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:43,  2.58it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:00,  4.41it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:57,  4.63it/s]
Loading 0:  10%|▉         | 28/291 [00:06<00:50,  5.17it/s]
Loading 0:  10%|█         | 30/291 [00:07<01:00,  4.31it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:20,  3.25it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:41,  2.56it/s]
Loading 0:  12%|█▏        | 35/291 [00:09<00:58,  4.36it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:55,  4.59it/s]
Loading 0:  13%|█▎        | 37/291 [00:09<00:49,  5.16it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:58,  4.31it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:17,  3.25it/s]
Loading 0:  14%|█▍        | 41/291 [00:11<01:37,  2.57it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:56,  4.37it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:53,  4.62it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<00:53,  4.52it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:08,  3.51it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:26,  2.79it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:53,  4.46it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:50,  4.68it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:51,  4.51it/s]
Loading 0:  20%|█▉        | 58/291 [00:15<01:05,  3.53it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:22,  2.80it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:51,  4.44it/s]
Loading 0:  22%|██▏       | 63/291 [00:16<00:48,  4.66it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:49,  4.53it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:03,  3.53it/s]
Loading 0:  23%|██▎       | 68/291 [00:18<01:19,  2.79it/s]
Loading 0:  24%|██▍       | 71/291 [00:18<00:49,  4.48it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:46,  4.69it/s]
Loading 0:  26%|██▌       | 75/291 [00:19<00:47,  4.54it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<01:00,  3.54it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<01:16,  2.80it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:47,  4.49it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:44,  4.69it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:39,  5.26it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:38,  5.45it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:58,  3.54it/s]
Loading 0:  29%|██▉       | 85/291 [00:22<01:14,  2.77it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:29,  2.28it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:48,  4.17it/s]
Loading 0:  31%|███       | 90/291 [00:23<00:45,  4.45it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:44,  4.42it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<00:57,  3.43it/s]
Loading 0:  33%|███▎      | 95/291 [00:25<01:12,  2.72it/s]
Loading 0:  34%|███▎      | 98/291 [00:25<00:43,  4.39it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:41,  4.62it/s]
Loading 0:  35%|███▌      | 102/291 [00:26<00:41,  4.51it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:53,  3.51it/s]
Loading 0:  36%|███▌      | 104/291 [00:27<01:07,  2.78it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:41,  4.40it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:39,  4.63it/s]
Loading 0:  37%|███▋      | 109/291 [00:27<00:35,  5.19it/s]
Loading 0:  38%|███▊      | 111/291 [00:28<00:41,  4.36it/s]
Loading 0:  38%|███▊      | 112/291 [00:29<00:54,  3.30it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:08,  2.61it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:39,  4.46it/s]
Loading 0:  40%|████      | 117/291 [00:30<00:37,  4.68it/s]
Loading 0:  41%|████      | 118/291 [00:30<00:32,  5.26it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:39,  4.35it/s]
Loading 0:  42%|████▏     | 121/291 [00:31<00:51,  3.27it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<01:05,  2.60it/s]
Loading 0:  43%|████▎     | 125/291 [00:32<00:37,  4.47it/s]
Loading 0:  43%|████▎     | 126/291 [00:32<00:35,  4.70it/s]
Loading 0:  44%|████▎     | 127/291 [00:32<00:31,  5.27it/s]
Loading 0:  44%|████▍     | 129/291 [00:33<00:37,  4.37it/s]
Loading 0:  45%|████▍     | 130/291 [00:33<00:49,  3.28it/s]
Loading 0:  45%|████▌     | 131/291 [00:34<01:01,  2.59it/s]
Loading 0:  46%|████▌     | 134/291 [00:34<00:35,  4.46it/s]
Loading 0:  46%|████▋     | 135/291 [00:34<00:33,  4.68it/s]
Loading 0:  47%|████▋     | 136/291 [00:34<00:29,  5.24it/s]
Loading 0:  47%|████▋     | 138/291 [00:35<00:35,  4.35it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:46,  3.28it/s]
Loading 0:  48%|████▊     | 140/291 [00:36<00:58,  2.60it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:33,  4.41it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:31,  4.64it/s]
Loading 0:  50%|████▉     | 145/291 [00:37<00:27,  5.24it/s]
Loading 0:  51%|█████     | 147/291 [00:37<00:33,  4.36it/s]
Loading 0:  51%|█████     | 148/291 [00:38<00:43,  3.28it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:55,  2.57it/s]
Loading 0:  52%|█████▏    | 152/291 [00:39<00:31,  4.37it/s]
Loading 0:  53%|█████▎    | 153/291 [00:39<00:29,  4.61it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:30,  4.50it/s]
Loading 0:  54%|█████▍    | 157/291 [00:40<00:38,  3.50it/s]
Loading 0:  54%|█████▍    | 158/291 [00:41<00:47,  2.78it/s]
Loading 0:  55%|█████▌    | 161/291 [00:41<00:29,  4.43it/s]
Loading 0:  56%|█████▌    | 162/291 [00:41<00:27,  4.65it/s]
Loading 0:  57%|█████▋    | 165/291 [00:42<00:27,  4.53it/s]
Loading 0:  57%|█████▋    | 166/291 [00:42<00:35,  3.54it/s]
Loading 0:  57%|█████▋    | 167/291 [00:43<00:44,  2.81it/s]
Loading 0:  58%|█████▊    | 170/291 [00:43<00:26,  4.51it/s]
Loading 0:  59%|█████▉    | 171/291 [00:43<00:25,  4.72it/s]
Loading 0:  59%|█████▉    | 173/291 [00:44<00:30,  3.91it/s]
Loading 0:  60%|██████    | 175/291 [00:44<00:23,  4.92it/s]
Loading 0:  60%|██████    | 176/291 [00:44<00:22,  5.08it/s]
Loading 0:  61%|██████    | 177/291 [00:44<00:20,  5.59it/s]
Loading 0:  62%|██████▏   | 179/291 [00:45<00:24,  4.53it/s]
Loading 0:  62%|██████▏   | 180/291 [00:46<00:32,  3.38it/s]
Loading 0:  62%|██████▏   | 181/291 [00:46<00:41,  2.67it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:23,  4.58it/s]
Loading 0:  64%|██████▎   | 185/291 [00:47<00:22,  4.81it/s]
Loading 0:  64%|██████▍   | 186/291 [00:47<00:19,  5.39it/s]
Loading 0:  64%|██████▍   | 187/291 [00:47<00:19,  5.39it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:29,  3.47it/s]
Loading 0:  65%|██████▍   | 189/291 [00:48<00:39,  2.61it/s]
Loading 0:  66%|██████▌   | 192/291 [00:49<00:27,  3.55it/s]
Loading 0:  66%|██████▋   | 193/291 [00:49<00:33,  2.94it/s]
Loading 0:  67%|██████▋   | 194/291 [00:50<00:39,  2.46it/s]
Loading 0:  68%|██████▊   | 197/291 [00:50<00:22,  4.13it/s]
Loading 0:  68%|██████▊   | 198/291 [00:50<00:21,  4.39it/s]
Loading 0:  68%|██████▊   | 199/291 [00:50<00:18,  4.92it/s]
Loading 0:  69%|██████▉   | 201/291 [00:51<00:21,  4.25it/s]
Loading 0:  69%|██████▉   | 202/291 [00:52<00:27,  3.25it/s]
Loading 0:  70%|██████▉   | 203/291 [00:52<00:33,  2.62it/s]
Loading 0:  71%|███████   | 206/291 [00:52<00:19,  4.43it/s]
Loading 0:  71%|███████   | 207/291 [00:53<00:17,  4.68it/s]
Loading 0:  71%|███████▏  | 208/291 [00:53<00:15,  5.28it/s]
Loading 0:  72%|███████▏  | 210/291 [00:53<00:18,  4.43it/s]
Loading 0:  73%|███████▎  | 211/291 [00:54<00:24,  3.33it/s]
Loading 0:  73%|███████▎  | 212/291 [00:54<00:29,  2.65it/s]
Loading 0:  74%|███████▍  | 215/291 [00:55<00:16,  4.55it/s]
Loading 0:  74%|███████▍  | 216/291 [00:55<00:15,  4.86it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:15,  4.69it/s]
Loading 0:  76%|███████▌  | 220/291 [00:56<00:19,  3.62it/s]
Loading 0:  76%|███████▌  | 221/291 [00:57<00:24,  2.88it/s]
Loading 0:  77%|███████▋  | 224/291 [00:57<00:14,  4.65it/s]
Loading 0:  77%|███████▋  | 225/291 [00:57<00:13,  4.86it/s]
Loading 0:  78%|███████▊  | 226/291 [00:57<00:12,  5.40it/s]
Loading 0:  78%|███████▊  | 228/291 [00:58<00:14,  4.49it/s]
Loading 0:  79%|███████▊  | 229/291 [00:58<00:18,  3.39it/s]
Loading 0:  79%|███████▉  | 230/291 [00:59<00:22,  2.69it/s]
Loading 0:  80%|████████  | 233/291 [00:59<00:12,  4.54it/s]
Loading 0:  80%|████████  | 234/291 [00:59<00:11,  4.78it/s]
Loading 0:  81%|████████  | 235/291 [00:59<00:10,  5.37it/s]
Loading 0:  81%|████████▏ | 237/291 [01:00<00:12,  4.47it/s]
Loading 0:  82%|████████▏ | 238/291 [01:00<00:15,  3.37it/s]
Loading 0:  82%|████████▏ | 239/291 [01:01<00:19,  2.67it/s]
Loading 0:  83%|████████▎ | 242/291 [01:01<00:10,  4.53it/s]
Loading 0:  84%|████████▎ | 243/291 [01:01<00:10,  4.76it/s]
Loading 0:  84%|████████▍ | 244/291 [01:02<00:08,  5.32it/s]
Loading 0:  85%|████████▍ | 246/291 [01:02<00:10,  4.43it/s]
Loading 0:  85%|████████▍ | 247/291 [01:03<00:13,  3.35it/s]
Loading 0:  85%|████████▌ | 248/291 [01:03<00:16,  2.67it/s]
Loading 0:  86%|████████▋ | 251/291 [01:03<00:08,  4.59it/s]
Loading 0:  87%|████████▋ | 252/291 [01:04<00:08,  4.82it/s]
Loading 0:  87%|████████▋ | 253/291 [01:04<00:07,  5.42it/s]
Loading 0:  88%|████████▊ | 255/291 [01:04<00:08,  4.47it/s]
Loading 0:  88%|████████▊ | 256/291 [01:05<00:10,  3.35it/s]
Loading 0:  88%|████████▊ | 257/291 [01:06<00:12,  2.66it/s]
Loading 0:  89%|████████▉ | 260/291 [01:06<00:06,  4.58it/s]
Loading 0:  90%|████████▉ | 261/291 [01:06<00:06,  4.82it/s]
Loading 0:  90%|█████████ | 262/291 [01:06<00:05,  5.42it/s]
Loading 0:  91%|█████████ | 264/291 [01:07<00:06,  4.45it/s]
Loading 0:  91%|█████████ | 265/291 [01:07<00:07,  3.36it/s]
Loading 0:  91%|█████████▏| 266/291 [01:08<00:09,  2.67it/s]
Loading 0:  92%|█████████▏| 269/291 [01:08<00:04,  4.54it/s]
Loading 0:  93%|█████████▎| 270/291 [01:08<00:04,  4.78it/s]
Loading 0:  93%|█████████▎| 271/291 [01:08<00:03,  5.39it/s]
Loading 0:  94%|█████████▍| 273/291 [01:09<00:04,  4.45it/s]
Loading 0:  94%|█████████▍| 274/291 [01:09<00:05,  3.35it/s]
Loading 0:  95%|█████████▍| 275/291 [01:10<00:06,  2.65it/s]
Loading 0:  96%|█████████▌| 278/291 [01:10<00:02,  4.56it/s]
Loading 0:  96%|█████████▌| 279/291 [01:10<00:02,  4.79it/s]
Loading 0:  96%|█████████▌| 280/291 [01:10<00:02,  5.38it/s]
Loading 0:  97%|█████████▋| 281/291 [01:11<00:02,  3.62it/s]
Loading 0:  97%|█████████▋| 282/291 [01:12<00:03,  2.72it/s]
Loading 0:  98%|█████████▊| 284/291 [01:12<00:01,  3.95it/s]
Loading 0:  98%|█████████▊| 285/291 [01:12<00:01,  4.30it/s]
Loading 0:  98%|█████████▊| 286/291 [01:12<00:01,  4.94it/s]
Loading 0:  99%|█████████▊| 287/291 [01:12<00:00,  4.88it/s]
Loading 0:  99%|█████████▉| 288/291 [01:13<00:00,  3.28it/s]
                                                            
Job rirv938-llama-8b-big-ret-4805-v1-mkmlizer completed after 143.72s with status: succeeded
Stopping job with name rirv938-llama-8b-big-ret-4805-v1-mkmlizer
Pipeline stage MKMLizer completed in 144.63s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-big-ret-4805-v1
Waiting for inference service rirv938-llama-8b-big-ret-4805-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service rirv938-llama-8b-big-ret-4805-v1 ready after 220.79706025123596s
Pipeline stage MKMLDeployer completed in 221.23s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 5.880456209182739s
Received healthy response to inference request in 4.690423965454102s
Received healthy response to inference request in 4.2860329151153564s
Received healthy response to inference request in 4.3659398555755615s
Received healthy response to inference request in 4.629454612731934s
5 requests
0 failed requests
5th percentile: 4.302014303207398
10th percentile: 4.317995691299439
20th percentile: 4.34995846748352
30th percentile: 4.418642807006836
40th percentile: 4.524048709869385
50th percentile: 4.629454612731934
60th percentile: 4.653842353820801
70th percentile: 4.678230094909668
80th percentile: 4.92843041419983
90th percentile: 5.404443311691284
95th percentile: 5.642449760437011
99th percentile: 5.8328549194335935
mean time: 4.770461511611939
%s, retrying in %s seconds...
Received healthy response to inference request in 5.730457544326782s
Received healthy response to inference request in 5.295873165130615s
Received healthy response to inference request in 5.07240629196167s
Received healthy response to inference request in 5.508867502212524s
Received healthy response to inference request in 4.020166397094727s
5 requests
0 failed requests
5th percentile: 4.230614376068115
10th percentile: 4.441062355041504
20th percentile: 4.861958312988281
30th percentile: 5.117099666595459
40th percentile: 5.206486415863037
50th percentile: 5.295873165130615
60th percentile: 5.381070899963379
70th percentile: 5.466268634796142
80th percentile: 5.553185510635376
90th percentile: 5.6418215274810795
95th percentile: 5.686139535903931
99th percentile: 5.721593942642212
mean time: 5.125554180145263
%s, retrying in %s seconds...
Received healthy response to inference request in 3.7438478469848633s
Received healthy response to inference request in 6.179864406585693s
Received healthy response to inference request in 4.575886249542236s
Received healthy response to inference request in 5.019899368286133s
Received healthy response to inference request in 4.80572509765625s
5 requests
0 failed requests
5th percentile: 3.910255527496338
10th percentile: 4.076663208007813
20th percentile: 4.409478569030762
30th percentile: 4.621854019165039
40th percentile: 4.713789558410644
50th percentile: 4.80572509765625
60th percentile: 4.891394805908203
70th percentile: 4.977064514160157
80th percentile: 5.251892375946045
90th percentile: 5.7158783912658695
95th percentile: 5.947871398925781
99th percentile: 6.133465805053711
mean time: 4.865044593811035
clean up pipeline due to error=%s
Shutdown handler de-registered
rirv938-llama-8b-big-ret_4805_v1 status is now failed due to DeploymentManager action
rirv938-llama-8b-big-ret_4805_v1 status is now torndown due to DeploymentManager action