rirv938-llama-8b-big-ret_4805

submission_id: rirv938-llama-8b-big-ret_4805_v2
developer_uid: robert_irvine
best_of: 1
display_name: rirv938-llama-8b-big-ret_4805_v2
family_friendly_score: 0.0
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: rirv938/llama_8b_big_retune_6m_4392
max_input_tokens: 256
max_output_tokens: 1
model_architecture: LlamaForSequenceClassification
model_group: rirv938/llama_8b_big_ret
model_name: rirv938-llama-8b-big-ret_4805_v2
model_num_parameters: 8030261248.0
model_repo: rirv938/llama_8b_big_retune_6m_4392
model_size: 8B
num_battles: 0
num_wins: 0
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-09-25T18:01:12+00:00
us_pacific_date: 2024-09-25
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-big-ret-4805-v2-mkmlizer
Waiting for job on rirv938-llama-8b-big-ret-4805-v2-mkmlizer to finish
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║     _____            __           __                                ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║       /___/                                                         ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  belonging to:                                                      ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: Downloaded to shared memory in 20.940s
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmph5km4sox, device:0
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: quantized model in 85.289s
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: Processed model rirv938/llama_8b_big_retune_6m_4392 in 106.230s
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v2
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v2/config.json
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v2/special_tokens_map.json
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v2/tokenizer_config.json
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4805-v2/flywheel_model.0.safetensors
rirv938-llama-8b-big-ret-4805-v2-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:56,  5.13it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:31,  3.14it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:00,  2.37it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:02,  4.52it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:01,  4.57it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:53,  5.21it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:04,  4.31it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:25,  3.24it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:49,  2.54it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:02,  4.37it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:59,  4.61it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:52,  5.21it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:01,  4.36it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:21,  3.29it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:43,  2.58it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:00,  4.40it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:56,  4.65it/s]
Loading 0:  10%|█         | 30/291 [00:07<00:57,  4.55it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:13,  3.53it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:32,  2.79it/s]
Loading 0:  12%|█▏        | 35/291 [00:08<00:56,  4.52it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:53,  4.73it/s]
Loading 0:  13%|█▎        | 37/291 [00:09<00:47,  5.31it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:56,  4.42it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:15,  3.34it/s]
Loading 0:  14%|█▍        | 41/291 [00:11<01:35,  2.61it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:55,  4.41it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:52,  4.65it/s]
Loading 0:  16%|█▌        | 47/291 [00:11<00:38,  6.38it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:12,  3.36it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:28,  2.72it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:54,  4.36it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:51,  4.58it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:51,  4.51it/s]
Loading 0:  20%|█▉        | 58/291 [00:14<01:06,  3.53it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:22,  2.82it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:51,  4.46it/s]
Loading 0:  22%|██▏       | 63/291 [00:15<00:48,  4.69it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:49,  4.58it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:02,  3.58it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:18,  2.85it/s]
Loading 0:  24%|██▍       | 71/291 [00:18<00:48,  4.51it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:46,  4.73it/s]
Loading 0:  26%|██▌       | 75/291 [00:18<00:46,  4.60it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<00:59,  3.59it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<01:15,  2.84it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:46,  4.50it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:44,  4.72it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:39,  5.28it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:39,  5.30it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:58,  3.55it/s]
Loading 0:  29%|██▉       | 85/291 [00:21<01:13,  2.79it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:28,  2.31it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:47,  4.21it/s]
Loading 0:  31%|███       | 90/291 [00:22<00:44,  4.49it/s]
Loading 0:  32%|███▏      | 92/291 [00:22<00:31,  6.27it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<00:59,  3.30it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:12,  2.70it/s]
Loading 0:  34%|███▎      | 98/291 [00:25<00:44,  4.30it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:42,  4.54it/s]
Loading 0:  35%|███▌      | 102/291 [00:25<00:42,  4.49it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:53,  3.53it/s]
Loading 0:  36%|███▌      | 104/291 [00:27<01:06,  2.82it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:41,  4.47it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:39,  4.69it/s]
Loading 0:  38%|███▊      | 110/291 [00:27<00:28,  6.33it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:52,  3.39it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:04,  2.77it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:40,  4.33it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:38,  4.56it/s]
Loading 0:  41%|████      | 118/291 [00:29<00:33,  5.11it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:39,  4.32it/s]
Loading 0:  42%|████▏     | 121/291 [00:30<00:51,  3.31it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<01:04,  2.64it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:37,  4.47it/s]
Loading 0:  43%|████▎     | 126/291 [00:31<00:35,  4.71it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:35,  4.58it/s]
Loading 0:  45%|████▍     | 130/291 [00:33<00:45,  3.55it/s]
Loading 0:  45%|████▌     | 131/291 [00:33<00:56,  2.81it/s]
Loading 0:  46%|████▌     | 134/291 [00:34<00:34,  4.53it/s]
Loading 0:  46%|████▋     | 135/291 [00:34<00:32,  4.74it/s]
Loading 0:  47%|████▋     | 136/291 [00:34<00:29,  5.31it/s]
Loading 0:  47%|████▋     | 138/291 [00:34<00:34,  4.44it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:45,  3.36it/s]
Loading 0:  48%|████▊     | 140/291 [00:36<00:56,  2.66it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:32,  4.53it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:30,  4.76it/s]
Loading 0:  50%|████▉     | 145/291 [00:36<00:27,  5.35it/s]
Loading 0:  51%|█████     | 147/291 [00:37<00:32,  4.44it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:42,  3.33it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:54,  2.62it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:30,  4.50it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:29,  4.73it/s]
Loading 0:  53%|█████▎    | 154/291 [00:38<00:25,  5.32it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:30,  4.42it/s]
Loading 0:  54%|█████▍    | 157/291 [00:40<00:41,  3.23it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:51,  2.58it/s]
Loading 0:  55%|█████▌    | 161/291 [00:40<00:29,  4.44it/s]
Loading 0:  56%|█████▌    | 162/291 [00:41<00:27,  4.68it/s]
Loading 0:  56%|█████▌    | 163/291 [00:41<00:24,  5.27it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:28,  4.40it/s]
Loading 0:  57%|█████▋    | 166/291 [00:42<00:37,  3.33it/s]
Loading 0:  57%|█████▋    | 167/291 [00:42<00:47,  2.63it/s]
Loading 0:  58%|█████▊    | 170/291 [00:43<00:27,  4.46it/s]
Loading 0:  59%|█████▉    | 171/291 [00:43<00:25,  4.70it/s]
Loading 0:  59%|█████▉    | 172/291 [00:43<00:22,  5.27it/s]
Loading 0:  59%|█████▉    | 173/291 [00:43<00:33,  3.56it/s]
Loading 0:  60%|██████    | 175/291 [00:44<00:23,  4.88it/s]
Loading 0:  60%|██████    | 176/291 [00:44<00:22,  5.08it/s]
Loading 0:  61%|██████    | 177/291 [00:44<00:19,  5.71it/s]
Loading 0:  62%|██████▏   | 179/291 [00:45<00:24,  4.53it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:33,  3.33it/s]
Loading 0:  62%|██████▏   | 181/291 [00:46<00:42,  2.61it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:23,  4.55it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:22,  4.78it/s]
Loading 0:  64%|██████▍   | 186/291 [00:46<00:19,  5.36it/s]
Loading 0:  64%|██████▍   | 187/291 [00:46<00:19,  5.39it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:29,  3.53it/s]
Loading 0:  65%|██████▍   | 189/291 [00:48<00:38,  2.65it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:27,  3.59it/s]
Loading 0:  66%|██████▋   | 193/291 [00:49<00:33,  2.94it/s]
Loading 0:  67%|██████▋   | 194/291 [00:49<00:39,  2.46it/s]
Loading 0:  68%|██████▊   | 197/291 [00:50<00:22,  4.15it/s]
Loading 0:  68%|██████▊   | 198/291 [00:50<00:21,  4.42it/s]
Loading 0:  69%|██████▉   | 201/291 [00:50<00:20,  4.44it/s]
Loading 0:  69%|██████▉   | 202/291 [00:51<00:25,  3.50it/s]
Loading 0:  70%|██████▉   | 203/291 [00:52<00:31,  2.82it/s]
Loading 0:  71%|███████   | 206/291 [00:52<00:18,  4.49it/s]
Loading 0:  71%|███████   | 207/291 [00:52<00:17,  4.71it/s]
Loading 0:  72%|███████▏  | 210/291 [00:53<00:17,  4.59it/s]
Loading 0:  73%|███████▎  | 211/291 [00:53<00:22,  3.58it/s]
Loading 0:  73%|███████▎  | 212/291 [00:54<00:27,  2.85it/s]
Loading 0:  74%|███████▍  | 215/291 [00:54<00:16,  4.50it/s]
Loading 0:  74%|███████▍  | 216/291 [00:54<00:15,  4.71it/s]
Loading 0:  75%|███████▍  | 217/291 [00:54<00:14,  5.25it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:16,  4.42it/s]
Loading 0:  76%|███████▌  | 220/291 [00:56<00:21,  3.35it/s]
Loading 0:  76%|███████▌  | 221/291 [00:56<00:26,  2.66it/s]
Loading 0:  77%|███████▋  | 224/291 [00:56<00:14,  4.53it/s]
Loading 0:  77%|███████▋  | 225/291 [00:56<00:13,  4.76it/s]
Loading 0:  78%|███████▊  | 226/291 [00:57<00:12,  5.35it/s]
Loading 0:  78%|███████▊  | 228/291 [00:57<00:14,  4.44it/s]
Loading 0:  79%|███████▊  | 229/291 [00:58<00:18,  3.35it/s]
Loading 0:  79%|███████▉  | 230/291 [00:58<00:23,  2.65it/s]
Loading 0:  80%|████████  | 233/291 [00:59<00:12,  4.53it/s]
Loading 0:  80%|████████  | 234/291 [00:59<00:11,  4.75it/s]
Loading 0:  81%|████████  | 235/291 [00:59<00:10,  5.33it/s]
Loading 0:  81%|████████▏ | 237/291 [00:59<00:12,  4.43it/s]
Loading 0:  82%|████████▏ | 238/291 [01:00<00:15,  3.32it/s]
Loading 0:  82%|████████▏ | 239/291 [01:01<00:19,  2.62it/s]
Loading 0:  83%|████████▎ | 242/291 [01:01<00:11,  4.45it/s]
Loading 0:  84%|████████▎ | 243/291 [01:01<00:10,  4.69it/s]
Loading 0:  84%|████████▍ | 244/291 [01:01<00:08,  5.28it/s]
Loading 0:  85%|████████▍ | 246/291 [01:02<00:10,  4.41it/s]
Loading 0:  85%|████████▍ | 247/291 [01:02<00:13,  3.32it/s]
Loading 0:  85%|████████▌ | 248/291 [01:03<00:16,  2.64it/s]
Loading 0:  86%|████████▋ | 251/291 [01:03<00:08,  4.52it/s]
Loading 0:  87%|████████▋ | 252/291 [01:03<00:08,  4.76it/s]
Loading 0:  87%|████████▋ | 253/291 [01:03<00:07,  5.34it/s]
Loading 0:  88%|████████▊ | 255/291 [01:04<00:08,  4.44it/s]
Loading 0:  88%|████████▊ | 256/291 [01:04<00:10,  3.34it/s]
Loading 0:  88%|████████▊ | 257/291 [01:05<00:12,  2.65it/s]
Loading 0:  89%|████████▉ | 260/291 [01:05<00:06,  4.55it/s]
Loading 0:  90%|████████▉ | 261/291 [01:05<00:06,  4.78it/s]
Loading 0:  90%|█████████ | 262/291 [01:06<00:05,  5.39it/s]
Loading 0:  91%|█████████ | 264/291 [01:06<00:06,  4.44it/s]
Loading 0:  91%|█████████ | 265/291 [01:07<00:07,  3.35it/s]
Loading 0:  91%|█████████▏| 266/291 [01:07<00:09,  2.62it/s]
Loading 0:  92%|█████████▏| 269/291 [01:08<00:04,  4.50it/s]
Loading 0:  93%|█████████▎| 270/291 [01:08<00:04,  4.73it/s]
Loading 0:  93%|█████████▎| 271/291 [01:08<00:03,  5.31it/s]
Loading 0:  94%|█████████▍| 273/291 [01:08<00:04,  4.42it/s]
Loading 0:  94%|█████████▍| 274/291 [01:09<00:05,  3.32it/s]
Loading 0:  95%|█████████▍| 275/291 [01:10<00:06,  2.62it/s]
Loading 0:  96%|█████████▌| 278/291 [01:10<00:02,  4.45it/s]
Loading 0:  96%|█████████▌| 279/291 [01:10<00:02,  4.69it/s]
Loading 0:  96%|█████████▌| 280/291 [01:10<00:02,  5.24it/s]
Loading 0:  97%|█████████▋| 281/291 [01:11<00:02,  3.56it/s]
Loading 0:  97%|█████████▋| 282/291 [01:11<00:03,  2.71it/s]
Loading 0:  98%|█████████▊| 284/291 [01:12<00:01,  3.87it/s]
Loading 0:  98%|█████████▊| 285/291 [01:12<00:01,  4.22it/s]
Loading 0:  98%|█████████▊| 286/291 [01:12<00:01,  4.90it/s]
Loading 0:  99%|█████████▊| 287/291 [01:12<00:00,  5.05it/s]
Loading 0:  99%|█████████▉| 288/291 [01:13<00:00,  3.33it/s]
                                                            
Job rirv938-llama-8b-big-ret-4805-v2-mkmlizer completed after 126.3s with status: succeeded
Stopping job with name rirv938-llama-8b-big-ret-4805-v2-mkmlizer
Pipeline stage MKMLizer completed in 127.45s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-big-ret-4805-v2
Waiting for inference service rirv938-llama-8b-big-ret-4805-v2 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Retrying (%r) after connection broken by '%r': %s
Inference service rirv938-llama-8b-big-ret-4805-v2 ready after 221.15359234809875s
Pipeline stage MKMLDeployer completed in 221.55s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 6.179023027420044s
Received healthy response to inference request in 5.151780843734741s
Received healthy response to inference request in 6.805479049682617s
Received healthy response to inference request in 7.16735315322876s
Received healthy response to inference request in 6.668871879577637s
5 requests
0 failed requests
5th percentile: 5.357229280471802
10th percentile: 5.562677717208862
20th percentile: 5.973574590682984
30th percentile: 6.2769927978515625
40th percentile: 6.4729323387146
50th percentile: 6.668871879577637
60th percentile: 6.723514747619629
70th percentile: 6.7781576156616214
80th percentile: 6.877853870391846
90th percentile: 7.022603511810303
95th percentile: 7.094978332519531
99th percentile: 7.152878189086914
mean time: 6.39450159072876
%s, retrying in %s seconds...
Received healthy response to inference request in 6.256846189498901s
Received healthy response to inference request in 5.420194864273071s
Received healthy response to inference request in 6.445151329040527s
Received healthy response to inference request in 6.240527629852295s
Received healthy response to inference request in 5.671836614608765s
5 requests
0 failed requests
5th percentile: 5.47052321434021
10th percentile: 5.520851564407349
20th percentile: 5.621508264541626
30th percentile: 5.785574817657471
40th percentile: 6.013051223754883
50th percentile: 6.240527629852295
60th percentile: 6.2470550537109375
70th percentile: 6.25358247756958
80th percentile: 6.294507217407227
90th percentile: 6.369829273223877
95th percentile: 6.407490301132202
99th percentile: 6.4376191234588624
mean time: 6.006911325454712
%s, retrying in %s seconds...
Received healthy response to inference request in 7.459374189376831s
Received healthy response to inference request in 6.2771148681640625s
Received healthy response to inference request in 7.303809642791748s
Received healthy response to inference request in 5.846786737442017s
Received healthy response to inference request in 6.323632717132568s
5 requests
0 failed requests
5th percentile: 5.932852363586425
10th percentile: 6.018917989730835
20th percentile: 6.191049242019654
30th percentile: 6.286418437957764
40th percentile: 6.305025577545166
50th percentile: 6.323632717132568
60th percentile: 6.71570348739624
70th percentile: 7.107774257659912
80th percentile: 7.334922552108765
90th percentile: 7.397148370742798
95th percentile: 7.4282612800598145
99th percentile: 7.453151607513428
mean time: 6.642143630981446
clean up pipeline due to error=%s
Shutdown handler de-registered
rirv938-llama-8b-big-ret_4805_v2 status is now failed due to DeploymentManager action
rirv938-llama-8b-big-ret_4805_v2 status is now torndown due to DeploymentManager action
Waiting for inference service blend-rofur-2024-10-03 to be ready
Ignoring service blend-rofur-2024-10-03 already deployed
Ignoring service blend-rofur-2024-10-03 already deployed
Waiting for inference service blend-rofur-2024-10-03 to be ready
rirv938-llama-8b-big-ret_4805_v2 status is now torndown due to DeploymentManager action