rirv938-llama-8b-multihe_2706

developer_uid: robert_irvine
submission_id: rirv938-llama-8b-multihe_2706_v1
model_name: rirv938-llama-8b-multihe_2706_v1
model_group: rirv938/llama_8b_multihe
status: torndown
timestamp: 2024-11-07T22:31:43+00:00
num_battles: 10723
num_wins: 5504
celo_rating: 1246.34
family_friendly_score: 0.5778
family_friendly_standard_error: 0.006984943235273999
submission_type: basic
model_repo: rirv938/llama_8b_multihead_57m_preference
model_architecture: LlamaForSequenceClassification
model_num_parameters: 8030261248.0
best_of: 1
max_input_tokens: 256
max_output_tokens: 1
display_name: rirv938-llama-8b-multihe_2706_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: rirv938/llama_8b_multihead_57m_preference
model_size: 8B
ranking_group: single
us_pacific_date: 2024-11-07
win_ratio: 0.5132891914576144
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 256, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-multihe-2706-v1-mkmlizer
Waiting for job on rirv938-llama-8b-multihe-2706-v1-mkmlizer to finish
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║     _____            __           __                                ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║       /___/                                                         ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  Version: 0.11.33                                                   ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ║                                                                     ║
rirv938-llama-8b-multihe-2706-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
rirv938-llama-8b-multihe-2706-v1-mkmlizer: Downloaded to shared memory in 107.888s
rirv938-llama-8b-multihe-2706-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpto65pk4k, device:0
rirv938-llama-8b-multihe-2706-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-llama-8b-multihe-2706-v1-mkmlizer: quantized model in 89.986s
rirv938-llama-8b-multihe-2706-v1-mkmlizer: Processed model rirv938/llama_8b_multihead_57m_preference in 197.875s
rirv938-llama-8b-multihe-2706-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-multihe-2706-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-multihe-2706-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-multihe-2706-v1
rirv938-llama-8b-multihe-2706-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-multihe-2706-v1/config.json
rirv938-llama-8b-multihe-2706-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-multihe-2706-v1/special_tokens_map.json
rirv938-llama-8b-multihe-2706-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-multihe-2706-v1/tokenizer_config.json
rirv938-llama-8b-multihe-2706-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-llama-8b-multihe-2706-v1/tokenizer.json
rirv938-llama-8b-multihe-2706-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/291 [00:00<00:59,  4.88it/s]
Loading 0:   1%|▏         | 4/291 [00:01<01:36,  2.98it/s]
Loading 0:   2%|▏         | 5/291 [00:01<02:09,  2.21it/s]
Loading 0:   3%|▎         | 8/291 [00:02<01:07,  4.22it/s]
Loading 0:   3%|▎         | 9/291 [00:02<01:05,  4.28it/s]
Loading 0:   3%|▎         | 10/291 [00:02<00:57,  4.86it/s]
Loading 0:   4%|▍         | 12/291 [00:03<01:08,  4.08it/s]
Loading 0:   4%|▍         | 13/291 [00:03<01:30,  3.06it/s]
Loading 0:   5%|▍         | 14/291 [00:04<01:54,  2.41it/s]
Loading 0:   6%|▌         | 17/291 [00:04<01:05,  4.22it/s]
Loading 0:   6%|▌         | 18/291 [00:04<01:00,  4.50it/s]
Loading 0:   7%|▋         | 19/291 [00:04<00:53,  5.10it/s]
Loading 0:   7%|▋         | 21/291 [00:05<01:04,  4.21it/s]
Loading 0:   8%|▊         | 22/291 [00:06<01:25,  3.15it/s]
Loading 0:   8%|▊         | 23/291 [00:06<01:49,  2.44it/s]
Loading 0:   9%|▉         | 26/291 [00:06<01:02,  4.22it/s]
Loading 0:   9%|▉         | 27/291 [00:07<00:58,  4.51it/s]
Loading 0:  10%|▉         | 28/291 [00:07<00:52,  5.00it/s]
Loading 0:  10%|█         | 30/291 [00:07<00:43,  6.07it/s]
Loading 0:  11%|█         | 31/291 [00:07<00:43,  5.98it/s]
Loading 0:  11%|█         | 32/291 [00:07<00:40,  6.40it/s]
Loading 0:  11%|█▏        | 33/291 [00:07<00:43,  5.87it/s]
Loading 0:  12%|█▏        | 34/291 [00:08<01:13,  3.49it/s]
Loading 0:  12%|█▏        | 35/291 [00:09<01:35,  2.68it/s]
Loading 0:  12%|█▏        | 36/291 [00:09<01:57,  2.17it/s]
Loading 0:  13%|█▎        | 39/291 [00:10<01:20,  3.13it/s]
Loading 0:  14%|█▎        | 40/291 [00:11<01:34,  2.64it/s]
Loading 0:  14%|█▍        | 41/291 [00:11<01:51,  2.25it/s]
Loading 0:  15%|█▌        | 44/291 [00:11<01:03,  3.88it/s]
Loading 0:  15%|█▌        | 45/291 [00:12<00:58,  4.19it/s]
Loading 0:  16%|█▌        | 46/291 [00:12<00:52,  4.67it/s]
Loading 0:  16%|█▋        | 48/291 [00:12<01:00,  4.03it/s]
Loading 0:  17%|█▋        | 49/291 [00:13<01:18,  3.09it/s]
Loading 0:  17%|█▋        | 50/291 [00:14<01:37,  2.47it/s]
Loading 0:  18%|█▊        | 53/291 [00:14<00:56,  4.25it/s]
Loading 0:  19%|█▊        | 54/291 [00:14<00:53,  4.47it/s]
Loading 0:  19%|█▉        | 55/291 [00:14<00:46,  5.03it/s]
Loading 0:  20%|█▉        | 57/291 [00:15<00:55,  4.19it/s]
Loading 0:  20%|█▉        | 58/291 [00:15<01:13,  3.16it/s]
Loading 0:  20%|██        | 59/291 [00:16<01:33,  2.49it/s]
Loading 0:  21%|██▏       | 62/291 [00:16<00:53,  4.29it/s]
Loading 0:  22%|██▏       | 63/291 [00:16<00:50,  4.51it/s]
Loading 0:  22%|██▏       | 64/291 [00:16<00:44,  5.09it/s]
Loading 0:  23%|██▎       | 66/291 [00:17<00:53,  4.23it/s]
Loading 0:  23%|██▎       | 67/291 [00:18<01:10,  3.18it/s]
Loading 0:  23%|██▎       | 68/291 [00:18<01:30,  2.48it/s]
Loading 0:  24%|██▍       | 71/291 [00:19<00:51,  4.27it/s]
Loading 0:  25%|██▍       | 72/291 [00:19<00:48,  4.49it/s]
Loading 0:  25%|██▌       | 73/291 [00:19<00:43,  5.06it/s]
Loading 0:  25%|██▌       | 74/291 [00:19<01:03,  3.39it/s]
Loading 0:  26%|██▌       | 75/291 [00:20<01:24,  2.56it/s]
Loading 0:  26%|██▋       | 77/291 [00:20<00:57,  3.72it/s]
Loading 0:  27%|██▋       | 78/291 [00:21<00:52,  4.03it/s]
Loading 0:  27%|██▋       | 79/291 [00:21<00:45,  4.64it/s]
Loading 0:  27%|██▋       | 80/291 [00:21<00:44,  4.70it/s]
Loading 0:  28%|██▊       | 81/291 [00:21<01:08,  3.05it/s]
Loading 0:  28%|██▊       | 82/291 [00:22<01:24,  2.47it/s]
Loading 0:  29%|██▊       | 83/291 [00:23<01:39,  2.08it/s]
Loading 0:  30%|██▉       | 86/291 [00:23<00:52,  3.92it/s]
Loading 0:  30%|██▉       | 87/291 [00:23<00:48,  4.19it/s]
Loading 0:  30%|███       | 88/291 [00:23<00:42,  4.80it/s]
Loading 0:  31%|███       | 90/291 [00:24<00:49,  4.04it/s]
Loading 0:  31%|███▏      | 91/291 [00:24<01:04,  3.08it/s]
Loading 0:  32%|███▏      | 92/291 [00:25<01:21,  2.45it/s]
Loading 0:  33%|███▎      | 95/291 [00:25<00:46,  4.21it/s]
Loading 0:  33%|███▎      | 96/291 [00:26<00:43,  4.43it/s]
Loading 0:  33%|███▎      | 97/291 [00:26<00:38,  4.98it/s]
Loading 0:  34%|███▍      | 99/291 [00:26<00:46,  4.16it/s]
Loading 0:  34%|███▍      | 100/291 [00:27<01:00,  3.14it/s]
Loading 0:  35%|███▍      | 101/291 [00:28<01:15,  2.50it/s]
Loading 0:  36%|███▌      | 104/291 [00:28<00:43,  4.26it/s]
Loading 0:  36%|███▌      | 105/291 [00:28<00:41,  4.49it/s]
Loading 0:  36%|███▋      | 106/291 [00:28<00:37,  4.98it/s]
Loading 0:  37%|███▋      | 108/291 [00:29<00:43,  4.16it/s]
Loading 0:  37%|███▋      | 109/291 [00:29<00:58,  3.13it/s]
Loading 0:  38%|███▊      | 110/291 [00:30<01:12,  2.49it/s]
Loading 0:  39%|███▉      | 113/291 [00:30<00:42,  4.23it/s]
Loading 0:  39%|███▉      | 114/291 [00:30<00:39,  4.46it/s]
Loading 0:  40%|███▉      | 115/291 [00:30<00:35,  5.01it/s]
Loading 0:  40%|███▉      | 116/291 [00:31<00:51,  3.38it/s]
Loading 0:  41%|████      | 118/291 [00:31<00:38,  4.55it/s]
Loading 0:  41%|████      | 119/291 [00:31<00:36,  4.77it/s]
Loading 0:  41%|████      | 120/291 [00:32<00:31,  5.41it/s]
Loading 0:  42%|████▏     | 122/291 [00:32<00:39,  4.27it/s]
Loading 0:  43%|████▎     | 125/291 [00:33<00:36,  4.59it/s]
Loading 0:  43%|████▎     | 126/291 [00:33<00:47,  3.47it/s]
Loading 0:  44%|████▎     | 127/291 [00:34<01:00,  2.72it/s]
Loading 0:  45%|████▍     | 130/291 [00:34<00:36,  4.36it/s]
Loading 0:  45%|████▌     | 131/291 [00:34<00:35,  4.57it/s]
Loading 0:  45%|████▌     | 132/291 [00:34<00:31,  5.08it/s]
Loading 0:  46%|████▌     | 133/291 [00:35<00:31,  5.05it/s]
Loading 0:  46%|████▌     | 134/291 [00:35<00:46,  3.37it/s]
Loading 0:  46%|████▋     | 135/291 [00:36<01:02,  2.51it/s]
Loading 0:  47%|████▋     | 138/291 [00:37<00:45,  3.38it/s]
Loading 0:  48%|████▊     | 139/291 [00:37<00:54,  2.80it/s]
Loading 0:  48%|████▊     | 140/291 [00:38<01:04,  2.34it/s]
Loading 0:  49%|████▉     | 143/291 [00:38<00:37,  3.94it/s]
Loading 0:  49%|████▉     | 144/291 [00:38<00:35,  4.18it/s]
Loading 0:  50%|████▉     | 145/291 [00:38<00:30,  4.72it/s]
Loading 0:  51%|█████     | 147/291 [00:39<00:35,  4.07it/s]
Loading 0:  51%|█████     | 148/291 [00:40<00:45,  3.12it/s]
Loading 0:  51%|█████     | 149/291 [00:40<00:56,  2.50it/s]
Loading 0:  52%|█████▏    | 152/291 [00:40<00:32,  4.23it/s]
Loading 0:  53%|█████▎    | 153/291 [00:41<00:30,  4.46it/s]
Loading 0:  53%|█████▎    | 154/291 [00:41<00:27,  5.00it/s]
Loading 0:  54%|█████▎    | 156/291 [00:41<00:32,  4.17it/s]
Loading 0:  54%|█████▍    | 157/291 [00:42<00:42,  3.15it/s]
Loading 0:  54%|█████▍    | 158/291 [00:43<00:53,  2.49it/s]
Loading 0:  55%|█████▌    | 161/291 [00:43<00:30,  4.22it/s]
Loading 0:  56%|█████▌    | 162/291 [00:43<00:28,  4.45it/s]
Loading 0:  56%|█████▌    | 163/291 [00:43<00:25,  4.99it/s]
Loading 0:  57%|█████▋    | 165/291 [00:44<00:30,  4.17it/s]
Loading 0:  57%|█████▋    | 166/291 [00:44<00:39,  3.15it/s]
Loading 0:  57%|█████▋    | 167/291 [00:45<00:49,  2.50it/s]
Loading 0:  58%|█████▊    | 170/291 [00:45<00:28,  4.25it/s]
Loading 0:  59%|█████▉    | 171/291 [00:45<00:26,  4.47it/s]
Loading 0:  59%|█████▉    | 172/291 [00:46<00:23,  5.04it/s]
Loading 0:  60%|█████▉    | 174/291 [00:46<00:27,  4.22it/s]
Loading 0:  60%|██████    | 175/291 [00:47<00:36,  3.19it/s]
Loading 0:  60%|██████    | 176/291 [00:47<00:45,  2.53it/s]
Loading 0:  62%|██████▏   | 179/291 [00:48<00:26,  4.30it/s]
Loading 0:  62%|██████▏   | 180/291 [00:48<00:24,  4.52it/s]
Loading 0:  62%|██████▏   | 181/291 [00:48<00:21,  5.10it/s]
Loading 0:  63%|██████▎   | 183/291 [00:48<00:17,  6.15it/s]
Loading 0:  63%|██████▎   | 184/291 [00:48<00:17,  6.05it/s]
Loading 0:  64%|██████▎   | 185/291 [00:48<00:16,  6.57it/s]
Loading 0:  64%|██████▍   | 186/291 [00:49<00:16,  6.18it/s]
Loading 0:  64%|██████▍   | 187/291 [00:49<00:28,  3.60it/s]
Loading 0:  65%|██████▍   | 188/291 [00:50<00:37,  2.74it/s]
Loading 0:  65%|██████▍   | 189/291 [00:50<00:45,  2.22it/s]
Loading 0:  66%|██████▌   | 192/291 [00:51<00:31,  3.19it/s]
Loading 0:  66%|██████▋   | 193/291 [00:52<00:36,  2.68it/s]
Loading 0:  67%|██████▋   | 194/291 [00:52<00:43,  2.25it/s]
Loading 0:  68%|██████▊   | 197/291 [00:53<00:24,  3.83it/s]
Loading 0:  68%|██████▊   | 198/291 [00:53<00:22,  4.10it/s]
Loading 0:  68%|██████▊   | 199/291 [00:53<00:20,  4.60it/s]
Loading 0:  69%|██████▉   | 201/291 [00:53<00:22,  4.02it/s]
Loading 0:  69%|██████▉   | 202/291 [00:54<00:28,  3.09it/s]
Loading 0:  70%|██████▉   | 203/291 [00:55<00:35,  2.47it/s]
Loading 0:  71%|███████   | 206/291 [00:55<00:20,  4.20it/s]
Loading 0:  71%|███████   | 207/291 [00:55<00:18,  4.44it/s]
Loading 0:  71%|███████▏  | 208/291 [00:55<00:16,  4.98it/s]
Loading 0:  72%|███████▏  | 210/291 [00:56<00:19,  4.18it/s]
Loading 0:  73%|███████▎  | 211/291 [00:56<00:25,  3.18it/s]
Loading 0:  73%|███████▎  | 212/291 [00:57<00:31,  2.50it/s]
Loading 0:  74%|███████▍  | 215/291 [00:57<00:17,  4.25it/s]
Loading 0:  74%|███████▍  | 216/291 [00:57<00:16,  4.48it/s]
Loading 0:  75%|███████▍  | 217/291 [00:58<00:14,  5.02it/s]
Loading 0:  75%|███████▌  | 219/291 [00:58<00:17,  4.22it/s]
Loading 0:  76%|███████▌  | 220/291 [00:59<00:22,  3.19it/s]
Loading 0:  76%|███████▌  | 221/291 [00:59<00:27,  2.54it/s]
Loading 0:  77%|███████▋  | 224/291 [01:00<00:15,  4.36it/s]
Loading 0:  77%|███████▋  | 225/291 [01:00<00:14,  4.58it/s]
Loading 0:  78%|███████▊  | 226/291 [01:00<00:12,  5.13it/s]
Loading 0:  78%|███████▊  | 227/291 [01:00<00:18,  3.43it/s]
Loading 0:  78%|███████▊  | 228/291 [01:01<00:24,  2.59it/s]
Loading 0:  79%|███████▉  | 230/291 [01:01<00:16,  3.76it/s]
Loading 0:  79%|███████▉  | 231/291 [01:02<00:14,  4.10it/s]
Loading 0:  80%|███████▉  | 232/291 [01:02<00:12,  4.78it/s]
Loading 0:  80%|████████  | 233/291 [01:02<00:11,  5.01it/s]
Loading 0:  80%|████████  | 234/291 [01:02<00:17,  3.25it/s]
Loading 0:  81%|████████▏ | 237/291 [01:03<00:13,  4.02it/s]
Loading 0:  82%|████████▏ | 238/291 [01:04<00:16,  3.12it/s]
Loading 0:  82%|████████▏ | 239/291 [01:04<00:20,  2.50it/s]
Loading 0:  83%|████████▎ | 242/291 [01:04<00:11,  4.21it/s]
Loading 0:  84%|████████▎ | 243/291 [01:05<00:10,  4.44it/s]
Loading 0:  84%|████████▍ | 244/291 [01:05<00:09,  4.95it/s]
Loading 0:  85%|████████▍ | 246/291 [01:05<00:10,  4.14it/s]
Loading 0:  85%|████████▍ | 247/291 [01:06<00:13,  3.16it/s]
Loading 0:  85%|████████▌ | 248/291 [01:07<00:17,  2.51it/s]
Loading 0:  86%|████████▋ | 251/291 [01:07<00:09,  4.27it/s]
Loading 0:  87%|████████▋ | 252/291 [01:07<00:08,  4.50it/s]
Loading 0:  87%|████████▋ | 253/291 [01:07<00:07,  5.03it/s]
Loading 0:  88%|████████▊ | 255/291 [01:08<00:08,  4.21it/s]
Loading 0:  88%|████████▊ | 256/291 [01:08<00:10,  3.19it/s]
Loading 0:  88%|████████▊ | 257/291 [01:09<00:13,  2.54it/s]
Loading 0:  89%|████████▉ | 260/291 [01:09<00:07,  4.31it/s]
Loading 0:  90%|████████▉ | 261/291 [01:09<00:06,  4.54it/s]
Loading 0:  90%|█████████ | 262/291 [01:09<00:05,  5.10it/s]
Loading 0:  91%|█████████ | 264/291 [01:10<00:06,  4.24it/s]
Loading 0:  91%|█████████ | 265/291 [01:11<00:08,  3.20it/s]
Loading 0:  91%|█████████▏| 266/291 [01:11<00:10,  2.48it/s]
Loading 0:  92%|█████████▏| 269/291 [01:12<00:05,  4.20it/s]
Loading 0:  93%|█████████▎| 270/291 [01:12<00:04,  4.44it/s]
Loading 0:  93%|█████████▎| 271/291 [01:12<00:04,  4.98it/s]
Loading 0:  94%|█████████▍| 273/291 [01:12<00:04,  4.20it/s]
Loading 0:  94%|█████████▍| 274/291 [01:13<00:05,  3.19it/s]
Loading 0:  95%|█████████▍| 275/291 [01:14<00:06,  2.54it/s]
Loading 0:  96%|█████████▌| 278/291 [01:14<00:03,  4.30it/s]
Loading 0:  96%|█████████▌| 279/291 [01:14<00:02,  4.52it/s]
Loading 0:  96%|█████████▌| 280/291 [01:14<00:02,  5.01it/s]
Loading 0:  97%|█████████▋| 281/291 [01:15<00:02,  3.41it/s]
Loading 0:  97%|█████████▋| 283/291 [01:15<00:01,  4.59it/s]
Loading 0:  98%|█████████▊| 284/291 [01:15<00:01,  4.78it/s]
Loading 0:  98%|█████████▊| 285/291 [01:15<00:01,  5.33it/s]
Loading 0:  98%|█████████▊| 286/291 [01:16<00:00,  5.25it/s]
Loading 0:  99%|█████████▊| 287/291 [01:16<00:01,  3.33it/s]
Loading 0:  99%|█████████▉| 288/291 [01:17<00:01,  2.49it/s]
                                                            
Job rirv938-llama-8b-multihe-2706-v1-mkmlizer completed after 226.88s with status: succeeded
Stopping job with name rirv938-llama-8b-multihe-2706-v1-mkmlizer
Pipeline stage MKMLizer completed in 227.97s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.21s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-multihe-2706-v1
Waiting for inference service rirv938-llama-8b-multihe-2706-v1 to be ready
Inference service rirv938-llama-8b-multihe-2706-v1 ready after 170.67565441131592s
Pipeline stage MKMLDeployer completed in 171.30s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.8779447078704834s
Received healthy response to inference request in 6.777456760406494s
Received healthy response to inference request in 1.8237061500549316s
Received healthy response to inference request in 2.478360176086426s
Received healthy response to inference request in 2.521169900894165s
5 requests
0 failed requests
5th percentile: 1.9546369552612304
10th percentile: 2.085567760467529
20th percentile: 2.347429370880127
30th percentile: 2.4869221210479737
40th percentile: 2.504046010971069
50th percentile: 2.521169900894165
60th percentile: 2.6638798236846926
70th percentile: 2.8065897464752196
80th percentile: 3.657847118377686
90th percentile: 5.21765193939209
95th percentile: 5.997554349899291
99th percentile: 6.621476278305053
mean time: 3.2957275390625
Pipeline stage StressChecker completed in 17.95s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.64s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.57s
Shutdown handler de-registered
rirv938-llama-8b-multihe_2706_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-llama-8b-multihe-2706-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.85s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-llama-8b-multihe-2706-v1-profiler
Waiting for inference service rirv938-llama-8b-multihe-2706-v1-profiler to be ready
Inference service rirv938-llama-8b-multihe-2706-v1-profiler ready after 40.13551330566406s
Pipeline stage MKMLProfilerDeployer completed in 40.42s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplov45p4:/code/chaiverse_profiler_1731022824 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplov45p4 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1731022824 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1731022824/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplov45p4:/code/chaiverse_profiler_1731022838 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplov45p4:/code/chaiverse_profiler_1731022839 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplov45p4 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1731022839 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1731022839/summary.json'
clean up pipeline due to error=ISVCScriptError('Command failed with error: Defaulted container "kserve-container" out of: kserve-container, queue-proxy\nUnable to use a TTY - input is not a terminal or the right kind of file\n\n  0%|          | 0/200 [00:00<?, ?it/s]\n  0%|          | 1/200 [00:30<1:39:37, 30.04s/it]\n  1%|          | 2/200 [01:00<1:39:06, 30.03s/it]\n  2%|▏         | 3/200 [01:30<1:38:36, 30.03s/it]\n  2%|▏         | 4/200 [02:00<1:38:06, 30.03s/it]\n  2%|▎         | 5/200 [02:30<1:37:36, 30.03s/it]\n  3%|▎         | 6/200 [03:00<1:37:06, 30.03s/it]\n  4%|▎         | 7/200 [03:30<1:36:36, 30.03s/it]\n  4%|▍         | 8/200 [04:00<1:36:06, 30.03s/it]\n  4%|▍         | 9/200 [04:30<1:35:36, 30.03s/it]\n  5%|▌         | 10/200 [05:00<1:35:06, 30.03s/it]\n  6%|▌         | 11/200 [05:30<1:34:36, 30.03s/it]\n  6%|▌         | 12/200 [06:00<1:34:05, 30.03s/it]\n  6%|▋         | 13/200 [06:30<1:33:35, 30.03s/it]\n  7%|▋         | 14/200 [07:00<1:33:05, 30.03s/it]\n  8%|▊         | 15/200 [07:30<1:32:35, 30.03s/it]\n  8%|▊         | 16/200 [08:00<1:32:05, 30.03s/it]\n  8%|▊         | 17/200 [08:30<1:31:35, 30.03s/it]\n  9%|▉         | 18/200 [09:00<1:31:05, 30.03s/it]\n 10%|▉         | 19/200 [09:30<1:30:35, 30.03s/it]\n 10%|█         | 20/200 [10:00<1:30:05, 30.03s/it]\n 10%|█         | 21/200 [10:30<1:29:35, 30.03s/it]\n 11%|█         | 22/200 [11:00<1:29:05, 30.03s/it]\n 12%|█▏        | 23/200 [11:30<1:28:35, 30.03s/it]\n 12%|█▏        | 24/200 [12:00<1:28:05, 30.03s/it]\n 12%|█▎        | 25/200 [12:30<1:27:35, 30.03s/it]\n 13%|█▎        | 26/200 [13:00<1:27:05, 30.03s/it]\n 14%|█▎        | 27/200 [13:30<1:26:35, 30.03s/it]\n 14%|█▍        | 28/200 [14:00<1:26:05, 30.03s/it]\n 14%|█▍        | 29/200 [14:30<1:25:35, 30.03s/it]\n 15%|█▌        | 30/200 [15:00<1:25:05, 30.03s/it]\n 16%|█▌        | 31/200 [15:30<1:24:35, 30.03s/it]\n 16%|█▌        | 32/200 [16:01<1:24:05, 30.03s/it]\n 16%|█▋        | 33/200 [16:31<1:23:35, 30.03s/it]\n 17%|█▋        | 34/200 [17:01<1:23:05, 30.03s/it]\n 18%|█▊        | 35/200 [17:31<1:22:35, 30.03s/it]\n 18%|█▊        | 36/200 [18:01<1:22:05, 30.03s/it]\n 18%|█▊        | 37/200 [18:31<1:21:34, 30.03s/it]\n 19%|█▉        | 38/200 [19:01<1:21:04, 30.03s/it]\n 20%|█▉        | 39/200 [19:31<1:20:33, 30.02s/it]\n 20%|██        | 40/200 [20:01<1:20:04, 30.03s/it]\n 20%|██        | 41/200 [20:31<1:19:34, 30.03s/it]\n 21%|██        | 42/200 [21:01<1:19:04, 30.03s/it]\n 22%|██▏       | 43/200 [21:31<1:18:33, 30.02s/it]\n 22%|██▏       | 44/200 [22:01<1:18:03, 30.02s/it]\n 22%|██▎       | 45/200 [22:31<1:17:33, 30.02s/it]\n 23%|██▎       | 46/200 [23:01<1:17:03, 30.02s/it]\n 24%|██▎       | 47/200 [23:31<1:16:33, 30.02s/it]\n 24%|██▍       | 48/200 [24:01<1:16:04, 30.03s/it]\n 24%|██▍       | 49/200 [24:31<1:15:33, 30.02s/it]\n 25%|██▌       | 50/200 [25:01<1:15:02, 30.02s/it]\n 26%|██▌       | 51/200 [25:31<1:14:32, 30.02s/it]\n 26%|██▌       | 52/200 [26:01<1:14:03, 30.02s/it]\n 26%|██▋       | 53/200 [26:31<1:13:33, 30.02s/it]\n 27%|██▋       | 54/200 [27:01<1:13:03, 30.03s/it]\n 28%|██▊       | 55/200 [27:31<1:12:33, 30.03s/it]\n 28%|██▊       | 56/200 [28:01<1:12:04, 30.03s/it]\n 28%|██▊       | 57/200 [28:31<1:11:34, 30.03s/it]\n 29%|██▉       | 58/200 [29:01<1:11:04, 30.03s/it]\n 30%|██▉       | 59/200 [29:31<1:10:34, 30.03s/it]\n 30%|███       | 60/200 [30:01<1:10:04, 30.03s/it]\n 30%|███       | 61/200 [30:31<1:09:34, 30.03s/it]\n 31%|███       | 62/200 [31:01<1:09:04, 30.03s/it]\n 32%|███▏      | 63/200 [31:31<1:08:34, 30.03s/it]\n 32%|███▏      | 64/200 [32:01<1:08:04, 30.03s/it]\n 32%|███▎      | 65/200 [32:31<1:07:34, 30.03s/it]\n 33%|███▎      | 66/200 [33:01<1:07:04, 30.03s/it]\n 34%|███▎      | 67/200 [33:31<1:06:34, 30.03s/it]\n 34%|███▍      | 68/200 [34:01<1:06:04, 30.03s/it]\n 34%|███▍      | 69/200 [34:32<1:05:34, 30.03s/it]\n 35%|███▌      | 70/200 [35:02<1:05:04, 30.03s/it]\n 36%|███▌      | 71/200 [35:02<45:27, 21.14s/it]  command terminated with exit code 137\n, output: waiting for startup of TargetModel(endpoint=\'localhost\', route=\'GPT-J-6B-lit-v2\', namespace=\'tenant-chaiml-guanaco\', max_characters=9999, reward=False, url_format=\'{endpoint}-predictor-default.{namespace}.knative.ord1.coreweave.cloud\')\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30)\nRequest failed with: (\'Connection aborted.\', RemoteDisconnected(\'Remote end closed connection without response\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e929f90>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e994ac0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e995570>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9964a0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99cf10>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99c550>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996dd0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e994970>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e997730>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9944c0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99d4b0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9958d0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e994af0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e995de0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9979a0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99d090>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e994e20>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996e00>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9949a0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9954b0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99de70>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99c5b0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99dab0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99c760>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996b00>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996c20>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9968c0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99d6f0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e290>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99d9f0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e997a00>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e997670>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e997460>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99c790>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e170>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99d6f0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9954b0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996290>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99d6f0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99d0f0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99f0d0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99eef0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996ad0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99f7f0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99f6d0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99cee0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99c700>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9952a0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996d70>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99c490>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99fbe0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e290>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99ef50>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9954b0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99fa30>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99dfc0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e950>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99ea40>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e7d0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99ea10>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e050>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99c5b0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99ecb0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e900730>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996560>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99ef80>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99dbd0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99dcf0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99cee0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e996860>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9009d0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99c490>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99cee0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99f070>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99f760>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e901450>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9010f0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99ee60>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e3b0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99cfd0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e1d0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e901900>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9000d0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99ff40>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e590>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99d3c0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e901390>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9003d0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e900940>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99dd80>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99fd90>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e901cc0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e900a00>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e9002b0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e900e20>: Failed to establish a new connection: [Errno 111] Connection refused\'))\nRequest failed with: HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f878e99e1a0>: Failed to establish a new connection: [Errno 111] Connection refused\'))\n')
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-llama-8b-multihe-2706-v1-profiler is running
Tearing down inference service rirv938-llama-8b-multihe-2706-v1-profiler
Service rirv938-llama-8b-multihe-2706-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.83s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-llama-8b-multihe-2706-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.95s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-llama-8b-multihe-2706-v1-profiler
Waiting for inference service rirv938-llama-8b-multihe-2706-v1-profiler to be ready
Inference service rirv938-llama-8b-multihe-2706-v1-profiler ready after 110.25788187980652s
Pipeline stage MKMLProfilerDeployer completed in 110.61s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplognhl9:/code/chaiverse_profiler_1731025113 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplognhl9 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1731025113 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1731025113/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplognhl9:/code/chaiverse_profiler_1731027200 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplognhl9:/code/chaiverse_profiler_1731027200 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-llama-8b-mul855943b9b2c05e364c7a7d212e5fc06f-deplognhl9 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1731027200 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1731027200/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-llama-8b-multihe-2706-v1-profiler is running
Tearing down inference service rirv938-llama-8b-multihe-2706-v1-profiler
Service rirv938-llama-8b-multihe-2706-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.86s
Shutdown handler de-registered
rirv938-llama-8b-multihe_2706_v1 status is now inactive due to auto deactivation removed underperforming models
rirv938-llama-8b-multihe_2706_v1 status is now torndown due to DeploymentManager action