rirv938-llama-8b-big-ret_4237

developer_uid: robert_irvine
submission_id: rirv938-llama-8b-big-ret_4237_v2
model_name: rirv938-llama-8b-big-ret_4237_v2
model_group: rirv938/llama_8b_big_ret
status: torndown
timestamp: 2024-09-25T17:51:05+00:00
num_battles: 2975
num_wins: 1453
celo_rating: 1246.31
family_friendly_score: 0.6
family_friendly_standard_error: 0.012167539467460629
submission_type: basic
model_repo: rirv938/llama_8b_big_retune_6m_732
model_architecture: LlamaForSequenceClassification
model_num_parameters: 8030261248.0
best_of: 1
max_input_tokens: 256
max_output_tokens: 1
display_name: rirv938-llama-8b-big-ret_4237_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: rirv938/llama_8b_big_retune_6m_732
model_size: 8B
ranking_group: single
us_pacific_date: 2024-09-25
win_ratio: 0.4884033613445378
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-big-ret-4237-v2-mkmlizer
Waiting for job on rirv938-llama-8b-big-ret-4237-v2-mkmlizer to finish
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║     _____            __           __                                ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║       /___/                                                         ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  belonging to:                                                      ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ║                                                                     ║
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: Downloaded to shared memory in 20.621s
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpgfptzrhx, device:0
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: quantized model in 84.743s
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: Processed model rirv938/llama_8b_big_retune_6m_732 in 105.365s
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v2
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v2/config.json
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v2/tokenizer_config.json
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v2/special_tokens_map.json
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v2/tokenizer.json
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-llama-8b-big-ret-4237-v2/flywheel_model.0.safetensors
rirv938-llama-8b-big-ret-4237-v2-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:55,  5.16it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:31,  3.13it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:03,  2.31it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:03,  4.44it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:02,  4.50it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:54,  5.15it/s]
Loading 0:   4%|▍         | 12/291 [00:02<01:04,  4.31it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:25,  3.24it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:47,  2.58it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:01,  4.43it/s]
Loading 0:   6%|▌         | 18/291 [00:04<00:58,  4.66it/s]
Loading 0:   7%|▋         | 21/291 [00:05<00:59,  4.54it/s]
Loading 0:   8%|▊         | 22/291 [00:05<01:16,  3.53it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:36,  2.78it/s]
Loading 0:   9%|▉         | 26/291 [00:06<00:59,  4.45it/s]
Loading 0:   9%|▉         | 27/291 [00:06<00:56,  4.64it/s]
Loading 0:  10%|▉         | 28/291 [00:06<00:51,  5.14it/s]
Loading 0:  10%|█         | 30/291 [00:07<00:59,  4.36it/s]
Loading 0:  11%|█         | 31/291 [00:08<01:18,  3.31it/s]
Loading 0:  11%|█         | 32/291 [00:08<01:38,  2.63it/s]
Loading 0:  12%|█▏        | 35/291 [00:08<00:57,  4.44it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<00:54,  4.68it/s]
Loading 0:  13%|█▎        | 39/291 [00:09<00:55,  4.56it/s]
Loading 0:  14%|█▎        | 40/291 [00:10<01:10,  3.55it/s]
Loading 0:  14%|█▍        | 41/291 [00:11<01:28,  2.81it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<00:54,  4.54it/s]
Loading 0:  15%|█▌        | 45/291 [00:11<00:51,  4.76it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<00:52,  4.60it/s]
Loading 0:  17%|█▋        | 49/291 [00:12<01:07,  3.57it/s]
Loading 0:  17%|█▋        | 50/291 [00:13<01:25,  2.83it/s]
Loading 0:  18%|█▊        | 53/291 [00:13<00:53,  4.47it/s]
Loading 0:  19%|█▊        | 54/291 [00:13<00:50,  4.69it/s]
Loading 0:  19%|█▉        | 55/291 [00:13<00:44,  5.25it/s]
Loading 0:  20%|█▉        | 57/291 [00:14<00:53,  4.41it/s]
Loading 0:  20%|█▉        | 58/291 [00:14<01:09,  3.34it/s]
Loading 0:  20%|██        | 59/291 [00:15<01:28,  2.63it/s]
Loading 0:  21%|██▏       | 62/291 [00:15<00:51,  4.47it/s]
Loading 0:  22%|██▏       | 63/291 [00:15<00:48,  4.70it/s]
Loading 0:  22%|██▏       | 64/291 [00:16<00:43,  5.20it/s]
Loading 0:  23%|██▎       | 66/291 [00:16<00:51,  4.37it/s]
Loading 0:  23%|██▎       | 67/291 [00:17<01:07,  3.30it/s]
Loading 0:  23%|██▎       | 68/291 [00:17<01:25,  2.61it/s]
Loading 0:  24%|██▍       | 71/291 [00:18<00:49,  4.42it/s]
Loading 0:  25%|██▍       | 72/291 [00:18<00:46,  4.67it/s]
Loading 0:  25%|██▌       | 74/291 [00:18<00:33,  6.41it/s]
Loading 0:  26%|██▌       | 76/291 [00:19<01:03,  3.37it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<01:18,  2.74it/s]
Loading 0:  27%|██▋       | 80/291 [00:20<00:48,  4.32it/s]
Loading 0:  28%|██▊       | 81/291 [00:20<00:46,  4.55it/s]
Loading 0:  28%|██▊       | 82/291 [00:20<00:41,  5.10it/s]
Loading 0:  29%|██▊       | 83/291 [00:20<00:40,  5.11it/s]
Loading 0:  29%|██▉       | 84/291 [00:21<00:59,  3.46it/s]
Loading 0:  29%|██▉       | 85/291 [00:21<01:14,  2.76it/s]
Loading 0:  30%|██▉       | 86/291 [00:22<01:29,  2.28it/s]
Loading 0:  31%|███       | 89/291 [00:22<00:48,  4.20it/s]
Loading 0:  31%|███       | 90/291 [00:22<00:44,  4.48it/s]
Loading 0:  31%|███▏      | 91/291 [00:23<00:39,  5.11it/s]
Loading 0:  32%|███▏      | 93/291 [00:23<00:46,  4.30it/s]
Loading 0:  32%|███▏      | 94/291 [00:24<01:00,  3.26it/s]
Loading 0:  33%|███▎      | 95/291 [00:24<01:15,  2.59it/s]
Loading 0:  34%|███▎      | 98/291 [00:25<00:43,  4.49it/s]
Loading 0:  34%|███▍      | 99/291 [00:25<00:40,  4.73it/s]
Loading 0:  35%|███▍      | 101/291 [00:25<00:29,  6.50it/s]
Loading 0:  35%|███▌      | 103/291 [00:26<00:55,  3.38it/s]
Loading 0:  36%|███▌      | 104/291 [00:27<01:08,  2.73it/s]
Loading 0:  37%|███▋      | 107/291 [00:27<00:42,  4.33it/s]
Loading 0:  37%|███▋      | 108/291 [00:27<00:40,  4.57it/s]
Loading 0:  37%|███▋      | 109/291 [00:27<00:35,  5.13it/s]
Loading 0:  38%|███▊      | 111/291 [00:28<00:41,  4.36it/s]
Loading 0:  38%|███▊      | 112/291 [00:28<00:53,  3.33it/s]
Loading 0:  39%|███▉      | 113/291 [00:29<01:07,  2.64it/s]
Loading 0:  40%|███▉      | 116/291 [00:29<00:39,  4.42it/s]
Loading 0:  40%|████      | 117/291 [00:29<00:37,  4.66it/s]
Loading 0:  41%|████      | 118/291 [00:29<00:33,  5.24it/s]
Loading 0:  41%|████      | 120/291 [00:30<00:38,  4.39it/s]
Loading 0:  42%|████▏     | 121/291 [00:30<00:51,  3.33it/s]
Loading 0:  42%|████▏     | 122/291 [00:31<01:04,  2.63it/s]
Loading 0:  43%|████▎     | 125/291 [00:31<00:36,  4.50it/s]
Loading 0:  43%|████▎     | 126/291 [00:31<00:34,  4.73it/s]
Loading 0:  44%|████▎     | 127/291 [00:32<00:30,  5.33it/s]
Loading 0:  44%|████▍     | 129/291 [00:32<00:36,  4.43it/s]
Loading 0:  45%|████▍     | 130/291 [00:33<00:48,  3.32it/s]
Loading 0:  45%|████▌     | 131/291 [00:33<01:00,  2.63it/s]
Loading 0:  46%|████▌     | 134/291 [00:34<00:34,  4.51it/s]
Loading 0:  46%|████▋     | 135/291 [00:34<00:32,  4.74it/s]
Loading 0:  47%|████▋     | 138/291 [00:34<00:33,  4.61it/s]
Loading 0:  48%|████▊     | 139/291 [00:35<00:42,  3.57it/s]
Loading 0:  48%|████▊     | 140/291 [00:36<00:53,  2.84it/s]
Loading 0:  49%|████▉     | 143/291 [00:36<00:32,  4.57it/s]
Loading 0:  49%|████▉     | 144/291 [00:36<00:30,  4.78it/s]
Loading 0:  50%|████▉     | 145/291 [00:36<00:27,  5.35it/s]
Loading 0:  51%|█████     | 147/291 [00:37<00:32,  4.45it/s]
Loading 0:  51%|█████     | 148/291 [00:37<00:42,  3.37it/s]
Loading 0:  51%|█████     | 149/291 [00:38<00:53,  2.64it/s]
Loading 0:  52%|█████▏    | 152/291 [00:38<00:31,  4.46it/s]
Loading 0:  53%|█████▎    | 153/291 [00:38<00:28,  4.77it/s]
Loading 0:  54%|█████▎    | 156/291 [00:39<00:29,  4.61it/s]
Loading 0:  54%|█████▍    | 157/291 [00:39<00:37,  3.58it/s]
Loading 0:  54%|█████▍    | 158/291 [00:40<00:46,  2.85it/s]
Loading 0:  55%|█████▌    | 161/291 [00:40<00:28,  4.61it/s]
Loading 0:  56%|█████▌    | 162/291 [00:40<00:26,  4.81it/s]
Loading 0:  56%|█████▌    | 163/291 [00:41<00:23,  5.39it/s]
Loading 0:  57%|█████▋    | 165/291 [00:41<00:28,  4.46it/s]
Loading 0:  57%|█████▋    | 166/291 [00:42<00:37,  3.36it/s]
Loading 0:  57%|█████▋    | 167/291 [00:42<00:46,  2.67it/s]
Loading 0:  58%|█████▊    | 170/291 [00:43<00:26,  4.51it/s]
Loading 0:  59%|█████▉    | 171/291 [00:43<00:25,  4.74it/s]
Loading 0:  59%|█████▉    | 172/291 [00:43<00:22,  5.31it/s]
Loading 0:  59%|█████▉    | 173/291 [00:43<00:32,  3.58it/s]
Loading 0:  60%|██████    | 175/291 [00:44<00:24,  4.81it/s]
Loading 0:  60%|██████    | 176/291 [00:44<00:22,  5.05it/s]
Loading 0:  62%|██████▏   | 179/291 [00:44<00:23,  4.75it/s]
Loading 0:  62%|██████▏   | 180/291 [00:45<00:31,  3.57it/s]
Loading 0:  62%|██████▏   | 181/291 [00:46<00:39,  2.81it/s]
Loading 0:  63%|██████▎   | 184/291 [00:46<00:23,  4.62it/s]
Loading 0:  64%|██████▎   | 185/291 [00:46<00:21,  4.83it/s]
Loading 0:  64%|██████▍   | 186/291 [00:46<00:19,  5.40it/s]
Loading 0:  64%|██████▍   | 187/291 [00:46<00:19,  5.38it/s]
Loading 0:  65%|██████▍   | 188/291 [00:47<00:28,  3.56it/s]
Loading 0:  65%|██████▍   | 189/291 [00:48<00:38,  2.67it/s]
Loading 0:  66%|██████▌   | 192/291 [00:48<00:27,  3.60it/s]
Loading 0:  66%|██████▋   | 193/291 [00:49<00:32,  2.98it/s]
Loading 0:  67%|██████▋   | 194/291 [00:49<00:39,  2.48it/s]
Loading 0:  68%|██████▊   | 197/291 [00:49<00:22,  4.19it/s]
Loading 0:  68%|██████▊   | 198/291 [00:50<00:20,  4.45it/s]
Loading 0:  69%|██████▉   | 201/291 [00:50<00:20,  4.46it/s]
Loading 0:  69%|██████▉   | 202/291 [00:51<00:25,  3.51it/s]
Loading 0:  70%|██████▉   | 203/291 [00:52<00:31,  2.83it/s]
Loading 0:  71%|███████   | 206/291 [00:52<00:18,  4.50it/s]
Loading 0:  71%|███████   | 207/291 [00:52<00:17,  4.72it/s]
Loading 0:  71%|███████▏  | 208/291 [00:52<00:15,  5.28it/s]
Loading 0:  72%|███████▏  | 210/291 [00:53<00:18,  4.44it/s]
Loading 0:  73%|███████▎  | 211/291 [00:53<00:23,  3.36it/s]
Loading 0:  73%|███████▎  | 212/291 [00:54<00:29,  2.67it/s]
Loading 0:  74%|███████▍  | 215/291 [00:54<00:16,  4.49it/s]
Loading 0:  74%|███████▍  | 216/291 [00:54<00:15,  4.73it/s]
Loading 0:  75%|███████▍  | 217/291 [00:54<00:13,  5.29it/s]
Loading 0:  75%|███████▌  | 219/291 [00:55<00:16,  4.44it/s]
Loading 0:  76%|███████▌  | 220/291 [00:55<00:21,  3.35it/s]
Loading 0:  76%|███████▌  | 221/291 [00:56<00:26,  2.65it/s]
Loading 0:  77%|███████▋  | 224/291 [00:56<00:14,  4.49it/s]
Loading 0:  77%|███████▋  | 225/291 [00:56<00:13,  4.72it/s]
Loading 0:  78%|███████▊  | 226/291 [00:56<00:12,  5.32it/s]
Loading 0:  78%|███████▊  | 228/291 [00:57<00:14,  4.41it/s]
Loading 0:  79%|███████▊  | 229/291 [00:58<00:18,  3.34it/s]
Loading 0:  79%|███████▉  | 230/291 [00:58<00:23,  2.64it/s]
Loading 0:  80%|████████  | 233/291 [00:58<00:12,  4.54it/s]
Loading 0:  80%|████████  | 234/291 [00:59<00:11,  4.78it/s]
Loading 0:  81%|████████▏ | 237/291 [00:59<00:11,  4.61it/s]
Loading 0:  82%|████████▏ | 238/291 [01:00<00:14,  3.57it/s]
Loading 0:  82%|████████▏ | 239/291 [01:01<00:18,  2.85it/s]
Loading 0:  83%|████████▎ | 242/291 [01:01<00:10,  4.60it/s]
Loading 0:  84%|████████▎ | 243/291 [01:01<00:09,  4.82it/s]
Loading 0:  84%|████████▍ | 244/291 [01:01<00:08,  5.39it/s]
Loading 0:  85%|████████▍ | 246/291 [01:02<00:10,  4.48it/s]
Loading 0:  85%|████████▍ | 247/291 [01:02<00:12,  3.39it/s]
Loading 0:  85%|████████▌ | 248/291 [01:03<00:15,  2.69it/s]
Loading 0:  86%|████████▋ | 251/291 [01:03<00:08,  4.60it/s]
Loading 0:  87%|████████▋ | 252/291 [01:03<00:08,  4.83it/s]
Loading 0:  88%|████████▊ | 255/291 [01:04<00:07,  4.67it/s]
Loading 0:  88%|████████▊ | 256/291 [01:04<00:09,  3.60it/s]
Loading 0:  88%|████████▊ | 257/291 [01:05<00:11,  2.86it/s]
Loading 0:  89%|████████▉ | 260/291 [01:05<00:06,  4.63it/s]
Loading 0:  90%|████████▉ | 261/291 [01:05<00:06,  4.84it/s]
Loading 0:  90%|█████████ | 262/291 [01:05<00:05,  5.41it/s]
Loading 0:  91%|█████████ | 264/291 [01:06<00:06,  4.48it/s]
Loading 0:  91%|█████████ | 265/291 [01:07<00:07,  3.38it/s]
Loading 0:  91%|█████████▏| 266/291 [01:07<00:09,  2.67it/s]
Loading 0:  92%|█████████▏| 269/291 [01:07<00:04,  4.51it/s]
Loading 0:  93%|█████████▎| 270/291 [01:08<00:04,  4.75it/s]
Loading 0:  94%|█████████▍| 273/291 [01:08<00:03,  4.60it/s]
Loading 0:  94%|█████████▍| 274/291 [01:09<00:04,  3.59it/s]
Loading 0:  95%|█████████▍| 275/291 [01:09<00:05,  2.86it/s]
Loading 0:  96%|█████████▌| 278/291 [01:10<00:02,  4.56it/s]
Loading 0:  96%|█████████▌| 279/291 [01:10<00:02,  4.79it/s]
Loading 0:  96%|█████████▌| 280/291 [01:10<00:02,  5.27it/s]
Loading 0:  97%|█████████▋| 281/291 [01:10<00:02,  3.62it/s]
Loading 0:  97%|█████████▋| 282/291 [01:11<00:03,  2.75it/s]
Loading 0:  98%|█████████▊| 284/291 [01:11<00:01,  3.95it/s]
Loading 0:  98%|█████████▊| 285/291 [01:11<00:01,  4.30it/s]
Loading 0:  98%|█████████▊| 286/291 [01:12<00:01,  4.97it/s]
Loading 0:  99%|█████████▊| 287/291 [01:12<00:00,  5.13it/s]
Loading 0:  99%|█████████▉| 288/291 [01:12<00:00,  3.38it/s]
                                                            
Job rirv938-llama-8b-big-ret-4237-v2-mkmlizer completed after 124.37s with status: succeeded
Stopping job with name rirv938-llama-8b-big-ret-4237-v2-mkmlizer
Pipeline stage MKMLizer completed in 126.19s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.22s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-big-ret-4237-v2
Waiting for inference service rirv938-llama-8b-big-ret-4237-v2 to be ready
Failed to get response for submission blend_masal_2024-09-19: ('http://zonemercy-virgo-edit-v1-1e5-v13-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:43162->127.0.0.1:8080: read: connection reset by peer\n')
Inference service rirv938-llama-8b-big-ret-4237-v2 ready after 220.77022171020508s
Pipeline stage MKMLDeployer completed in 221.21s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 6.441389083862305s
Received healthy response to inference request in 6.097635984420776s
Received healthy response to inference request in 5.564625024795532s
Received healthy response to inference request in 6.9597086906433105s
Received healthy response to inference request in 5.955855369567871s
5 requests
0 failed requests
5th percentile: 5.64287109375
10th percentile: 5.721117162704468
20th percentile: 5.877609300613403
30th percentile: 5.984211492538452
40th percentile: 6.040923738479615
50th percentile: 6.097635984420776
60th percentile: 6.235137224197388
70th percentile: 6.372638463973999
80th percentile: 6.545053005218506
90th percentile: 6.752380847930908
95th percentile: 6.856044769287109
99th percentile: 6.93897590637207
mean time: 6.203842830657959
%s, retrying in %s seconds...
Received healthy response to inference request in 5.834542751312256s
Received healthy response to inference request in 5.885401010513306s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 5.45223593711853s
Received healthy response to inference request in 6.334063768386841s
Received healthy response to inference request in 6.962356805801392s
5 requests
0 failed requests
5th percentile: 5.528697299957275
10th percentile: 5.60515866279602
20th percentile: 5.758081388473511
30th percentile: 5.844714403152466
40th percentile: 5.865057706832886
50th percentile: 5.885401010513306
60th percentile: 6.0648661136627195
70th percentile: 6.244331216812133
80th percentile: 6.4597223758697515
90th percentile: 6.711039590835571
95th percentile: 6.836698198318481
99th percentile: 6.937225084304809
mean time: 6.0937200546264645
%s, retrying in %s seconds...
Received healthy response to inference request in 6.733880996704102s
Received healthy response to inference request in 4.397340536117554s
Received healthy response to inference request in 6.973529815673828s
Received healthy response to inference request in 7.366533994674683s
Received healthy response to inference request in 5.249093055725098s
5 requests
0 failed requests
5th percentile: 4.567691040039063
10th percentile: 4.738041543960572
20th percentile: 5.078742551803589
30th percentile: 5.546050643920898
40th percentile: 6.1399658203125
50th percentile: 6.733880996704102
60th percentile: 6.829740524291992
70th percentile: 6.925600051879883
80th percentile: 7.052130651473999
90th percentile: 7.2093323230743405
95th percentile: 7.2879331588745115
99th percentile: 7.350813827514648
mean time: 6.144075679779053
clean up pipeline due to error=%s
Shutdown handler de-registered
rirv938-llama-8b-big-ret_4237_v2 status is now failed due to DeploymentManager action
rirv938-llama-8b-big-ret_4237_v2 status is now inactive due to auto deactivation removed underperforming models
rirv938-llama-8b-big-ret_4237_v2 status is now torndown due to DeploymentManager action