submission_id: zonemercy-viral-ss-v0-1e5ep2_v30
developer_uid: chai_backend_admin
best_of: 8
celo_rating: 1260.67
display_name: temp-3
family_friendly_score: 0.5416827355701246
family_friendly_standard_error: 0.006895600834931795
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
is_internal_developer: True
language_model: zonemercy/Viral-ss-v0-1e5ep2
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: zonemercy/Viral-ss-v0-1e
model_name: temp-3
model_num_parameters: 22247282688.0
model_repo: zonemercy/Viral-ss-v0-1e5ep2
model_size: 22B
num_battles: 5398
num_wins: 2790
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-10-04T22:47:49+00:00
us_pacific_date: 2024-10-04
win_ratio: 0.5168580955909596
Download Preference Data
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer
Waiting for job on zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer to finish
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║     _____            __           __                                ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║       /___/                                                         ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║                                                                     ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║                                                                     ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  belonging to:                                                      ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║                                                                     ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ║                                                                     ║
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: Downloaded to shared memory in 131.642s
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpk3r_j8en, device:0
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: quantized model in 57.030s
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: Processed model zonemercy/Viral-ss-v0-1e5ep2 in 188.672s
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v30
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v30/special_tokens_map.json
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v30/config.json
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v30/tokenizer_config.json
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v30/tokenizer.json
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v30/flywheel_model.1.safetensors
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v30/flywheel_model.0.safetensors
zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/507 [00:00<00:16, 30.54it/s]
Loading 0:   2%|▏         | 8/507 [00:00<00:20, 24.52it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:20, 24.47it/s]
Loading 0:   3%|▎         | 15/507 [00:00<00:23, 20.64it/s]
Loading 0:   4%|▎         | 19/507 [00:00<00:20, 24.16it/s]
Loading 0:   5%|▍         | 23/507 [00:00<00:18, 26.81it/s]
Loading 0:   5%|▌         | 26/507 [00:01<00:26, 18.11it/s]
Loading 0:   6%|▌         | 29/507 [00:01<00:25, 18.87it/s]
Loading 0:   6%|▋         | 32/507 [00:01<00:29, 15.85it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:23, 19.86it/s]
Loading 0:   8%|▊         | 40/507 [00:01<00:22, 20.37it/s]
Loading 0:   8%|▊         | 43/507 [00:02<00:22, 20.24it/s]
Loading 0:   9%|▉         | 46/507 [00:02<00:21, 21.17it/s]
Loading 0:  10%|▉         | 50/507 [00:02<00:18, 24.15it/s]
Loading 0:  10%|█         | 53/507 [00:02<00:27, 16.45it/s]
Loading 0:  11%|█         | 56/507 [00:02<00:26, 17.26it/s]
Loading 0:  12%|█▏        | 59/507 [00:03<00:29, 14.94it/s]
Loading 0:  13%|█▎        | 64/507 [00:03<00:22, 19.65it/s]
Loading 0:  13%|█▎        | 67/507 [00:03<00:21, 20.03it/s]
Loading 0:  14%|█▍        | 70/507 [00:03<00:21, 19.97it/s]
Loading 0:  14%|█▍        | 73/507 [00:03<00:20, 20.97it/s]
Loading 0:  15%|█▌        | 77/507 [00:03<00:17, 23.94it/s]
Loading 0:  16%|█▌        | 80/507 [00:04<00:25, 16.77it/s]
Loading 0:  16%|█▋        | 83/507 [00:04<00:26, 15.87it/s]
Loading 0:  17%|█▋        | 87/507 [00:04<00:21, 19.14it/s]
Loading 0:  18%|█▊        | 91/507 [00:04<00:18, 22.34it/s]
Loading 0:  19%|█▊        | 94/507 [00:04<00:19, 20.98it/s]
Loading 0:  19%|█▉        | 97/507 [00:04<00:19, 21.19it/s]
Loading 0:  20%|█▉        | 100/507 [00:05<00:23, 17.09it/s]
Loading 0:  21%|██        | 105/507 [00:05<00:18, 21.65it/s]
Loading 0:  21%|██▏       | 108/507 [00:05<00:27, 14.43it/s]
Loading 0:  22%|██▏       | 112/507 [00:05<00:22, 17.28it/s]
Loading 0:  23%|██▎       | 115/507 [00:05<00:21, 17.91it/s]
Loading 0:  23%|██▎       | 118/507 [00:06<00:19, 19.47it/s]
Loading 0:  24%|██▍       | 121/507 [00:06<00:19, 20.00it/s]
Loading 0:  24%|██▍       | 124/507 [00:06<00:19, 20.14it/s]
Loading 0:  25%|██▌       | 127/507 [00:06<00:17, 21.45it/s]
Loading 0:  26%|██▌       | 131/507 [00:06<00:15, 24.61it/s]
Loading 0:  26%|██▋       | 134/507 [00:06<00:22, 16.68it/s]
Loading 0:  27%|██▋       | 137/507 [00:07<00:20, 17.85it/s]
Loading 0:  28%|██▊       | 140/507 [00:07<00:24, 15.10it/s]
Loading 0:  29%|██▊       | 145/507 [00:07<00:18, 19.89it/s]
Loading 0:  29%|██▉       | 148/507 [00:07<00:17, 20.46it/s]
Loading 0:  30%|██▉       | 151/507 [00:07<00:17, 20.39it/s]
Loading 0:  30%|███       | 154/507 [00:07<00:16, 21.30it/s]
Loading 0:  31%|███       | 157/507 [00:07<00:15, 23.20it/s]
Loading 0:  32%|███▏      | 160/507 [00:08<00:20, 17.15it/s]
Loading 0:  32%|███▏      | 163/507 [00:08<00:20, 16.44it/s]
Loading 0:  33%|███▎      | 166/507 [00:08<00:19, 17.93it/s]
Loading 0:  33%|███▎      | 169/507 [00:08<00:18, 18.18it/s]
Loading 0:  34%|███▍      | 172/507 [00:08<00:16, 19.80it/s]
Loading 0:  35%|███▍      | 175/507 [00:09<00:16, 20.40it/s]
Loading 0:  35%|███▌      | 178/507 [00:09<00:16, 20.37it/s]
Loading 0:  36%|███▌      | 181/507 [00:09<00:15, 21.53it/s]
Loading 0:  36%|███▋      | 185/507 [00:09<00:13, 24.63it/s]
Loading 0:  37%|███▋      | 188/507 [00:09<00:19, 16.75it/s]
Loading 0:  38%|███▊      | 191/507 [00:09<00:18, 17.46it/s]
Loading 0:  38%|███▊      | 194/507 [00:10<00:20, 15.06it/s]
Loading 0:  39%|███▉      | 199/507 [00:10<00:15, 19.91it/s]
Loading 0:  40%|███▉      | 202/507 [00:10<00:14, 20.56it/s]
Loading 0:  40%|████      | 205/507 [00:10<00:14, 20.63it/s]
Loading 0:  41%|████      | 208/507 [00:10<00:13, 21.53it/s]
Loading 0:  42%|████▏     | 211/507 [00:10<00:12, 23.38it/s]
Loading 0:  42%|████▏     | 214/507 [00:11<00:16, 17.27it/s]
Loading 0:  43%|████▎     | 217/507 [00:11<00:18, 16.04it/s]
Loading 0:  43%|████▎     | 220/507 [00:11<00:16, 17.46it/s]
Loading 0:  44%|████▍     | 222/507 [00:11<00:16, 16.84it/s]
Loading 0:  45%|████▍     | 226/507 [00:11<00:13, 20.24it/s]
Loading 0:  45%|████▌     | 229/507 [00:11<00:13, 20.78it/s]
Loading 0:  46%|████▌     | 232/507 [00:11<00:13, 20.70it/s]
Loading 0:  46%|████▋     | 235/507 [00:12<00:12, 21.82it/s]
Loading 0:  47%|████▋     | 238/507 [00:12<00:11, 23.66it/s]
Loading 0:  48%|████▊     | 241/507 [00:12<00:15, 16.78it/s]
Loading 0:  48%|████▊     | 244/507 [00:12<00:16, 16.26it/s]
Loading 0:  49%|████▊     | 247/507 [00:12<00:14, 17.65it/s]
Loading 0:  49%|████▉     | 249/507 [00:12<00:15, 16.93it/s]
Loading 0:  50%|████▉     | 253/507 [00:13<00:12, 20.33it/s]
Loading 0:  50%|█████     | 256/507 [00:13<00:12, 20.82it/s]
Loading 0:  51%|█████     | 259/507 [00:13<00:11, 20.70it/s]
Loading 0:  52%|█████▏    | 262/507 [00:13<00:11, 21.57it/s]
Loading 0:  52%|█████▏    | 265/507 [00:13<00:10, 23.52it/s]
Loading 0:  53%|█████▎    | 268/507 [00:13<00:14, 16.87it/s]
Loading 0:  53%|█████▎    | 271/507 [00:14<00:14, 16.30it/s]
Loading 0:  54%|█████▍    | 274/507 [00:14<00:13, 17.42it/s]
Loading 0:  54%|█████▍    | 276/507 [00:14<00:14, 16.29it/s]
Loading 0:  55%|█████▌    | 280/507 [00:14<00:11, 19.37it/s]
Loading 0:  56%|█████▌    | 283/507 [00:14<00:11, 19.33it/s]
Loading 0:  56%|█████▋    | 286/507 [00:14<00:11, 19.22it/s]
Loading 0:  57%|█████▋    | 289/507 [00:14<00:10, 20.20it/s]
Loading 0:  58%|█████▊    | 292/507 [00:15<00:09, 21.99it/s]
Loading 0:  58%|█████▊    | 295/507 [00:15<00:13, 16.11it/s]
Loading 0:  59%|█████▊    | 297/507 [00:15<00:14, 14.33it/s]
Loading 0:  59%|█████▉    | 299/507 [00:30<00:14, 14.33it/s]
Loading 0:  59%|█████▉    | 300/507 [00:30<05:49,  1.69s/it]
Loading 0:  60%|█████▉    | 302/507 [00:31<04:32,  1.33s/it]
Loading 0:  60%|██████    | 306/507 [00:31<02:43,  1.23it/s]
Loading 0:  61%|██████    | 308/507 [00:31<02:08,  1.55it/s]
Loading 0:  61%|██████    | 310/507 [00:31<01:39,  1.98it/s]
Loading 0:  62%|██████▏   | 312/507 [00:31<01:17,  2.53it/s]
Loading 0:  62%|██████▏   | 316/507 [00:31<00:46,  4.10it/s]
Loading 0:  63%|██████▎   | 320/507 [00:32<00:30,  6.12it/s]
Loading 0:  64%|██████▎   | 323/507 [00:32<00:27,  6.76it/s]
Loading 0:  64%|██████▍   | 325/507 [00:32<00:23,  7.66it/s]
Loading 0:  65%|██████▍   | 328/507 [00:32<00:18,  9.67it/s]
Loading 0:  65%|██████▌   | 330/507 [00:32<00:16, 10.53it/s]
Loading 0:  66%|██████▌   | 334/507 [00:32<00:12, 14.12it/s]
Loading 0:  66%|██████▋   | 337/507 [00:33<00:10, 15.80it/s]
Loading 0:  67%|██████▋   | 340/507 [00:33<00:09, 16.79it/s]
Loading 0:  68%|██████▊   | 343/507 [00:33<00:09, 18.19it/s]
Loading 0:  68%|██████▊   | 346/507 [00:33<00:07, 20.62it/s]
Loading 0:  69%|██████▉   | 349/507 [00:33<00:09, 16.04it/s]
Loading 0:  69%|██████▉   | 351/507 [00:33<00:10, 14.19it/s]
Loading 0:  70%|███████   | 355/507 [00:34<00:08, 17.81it/s]
Loading 0:  71%|███████   | 358/507 [00:34<00:08, 18.28it/s]
Loading 0:  71%|███████   | 361/507 [00:34<00:07, 19.64it/s]
Loading 0:  72%|███████▏  | 364/507 [00:34<00:07, 20.19it/s]
Loading 0:  72%|███████▏  | 367/507 [00:34<00:06, 20.07it/s]
Loading 0:  73%|███████▎  | 370/507 [00:34<00:06, 21.37it/s]
Loading 0:  74%|███████▎  | 373/507 [00:34<00:05, 23.36it/s]
Loading 0:  74%|███████▍  | 376/507 [00:35<00:07, 17.42it/s]
Loading 0:  75%|███████▍  | 379/507 [00:35<00:07, 16.76it/s]
Loading 0:  75%|███████▌  | 382/507 [00:35<00:06, 18.25it/s]
Loading 0:  76%|███████▌  | 385/507 [00:35<00:06, 18.84it/s]
Loading 0:  77%|███████▋  | 388/507 [00:35<00:05, 20.36it/s]
Loading 0:  77%|███████▋  | 391/507 [00:35<00:05, 20.86it/s]
Loading 0:  78%|███████▊  | 394/507 [00:35<00:05, 20.81it/s]
Loading 0:  78%|███████▊  | 397/507 [00:36<00:04, 22.04it/s]
Loading 0:  79%|███████▉  | 401/507 [00:36<00:04, 25.24it/s]
Loading 0:  80%|███████▉  | 404/507 [00:36<00:06, 17.14it/s]
Loading 0:  80%|████████  | 407/507 [00:36<00:05, 18.24it/s]
Loading 0:  81%|████████  | 410/507 [00:36<00:06, 15.41it/s]
Loading 0:  82%|████████▏ | 415/507 [00:37<00:04, 20.17it/s]
Loading 0:  82%|████████▏ | 418/507 [00:37<00:04, 20.54it/s]
Loading 0:  83%|████████▎ | 421/507 [00:37<00:04, 20.32it/s]
Loading 0:  84%|████████▎ | 424/507 [00:37<00:03, 20.93it/s]
Loading 0:  84%|████████▍ | 427/507 [00:37<00:03, 22.89it/s]
Loading 0:  85%|████████▍ | 430/507 [00:37<00:04, 16.70it/s]
Loading 0:  85%|████████▌ | 433/507 [00:38<00:04, 16.21it/s]
Loading 0:  86%|████████▌ | 436/507 [00:38<00:04, 17.74it/s]
Loading 0:  86%|████████▋ | 438/507 [00:38<00:04, 16.96it/s]
Loading 0:  87%|████████▋ | 442/507 [00:38<00:03, 20.07it/s]
Loading 0:  88%|████████▊ | 445/507 [00:38<00:02, 20.77it/s]
Loading 0:  88%|████████▊ | 448/507 [00:38<00:02, 20.89it/s]
Loading 0:  89%|████████▉ | 451/507 [00:38<00:02, 22.15it/s]
Loading 0:  90%|████████▉ | 455/507 [00:39<00:02, 25.31it/s]
Loading 0:  90%|█████████ | 458/507 [00:39<00:02, 17.08it/s]
Loading 0:  91%|█████████ | 461/507 [00:39<00:02, 17.80it/s]
Loading 0:  92%|█████████▏| 464/507 [00:39<00:02, 15.08it/s]
Loading 0:  93%|█████████▎| 469/507 [00:39<00:01, 19.95it/s]
Loading 0:  93%|█████████▎| 472/507 [00:40<00:01, 20.60it/s]
Loading 0:  94%|█████████▎| 475/507 [00:40<00:01, 20.54it/s]
Loading 0:  94%|█████████▍| 478/507 [00:40<00:01, 21.61it/s]
Loading 0:  95%|█████████▌| 482/507 [00:40<00:01, 24.55it/s]
Loading 0:  96%|█████████▌| 485/507 [00:42<00:05,  4.16it/s]
Loading 0:  96%|█████████▌| 487/507 [00:42<00:04,  4.74it/s]
Loading 0:  97%|█████████▋| 491/507 [00:43<00:02,  6.94it/s]
Loading 0:  97%|█████████▋| 494/507 [00:43<00:01,  8.48it/s]
Loading 0:  98%|█████████▊| 497/507 [00:43<00:00, 10.45it/s]
Loading 0:  99%|█████████▊| 500/507 [00:43<00:00, 12.48it/s]
Loading 0:  99%|█████████▉| 503/507 [00:43<00:00, 14.02it/s]
Loading 0: 100%|█████████▉| 506/507 [00:43<00:00, 15.99it/s]
                                                            

Job zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer completed after 238.17s with status: succeeded
Stopping job with name zonemercy-viral-ss-v0-1e5ep2-v30-mkmlizer
Pipeline stage MKMLizer completed in 238.71s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-viral-ss-v0-1e5ep2-v30
Waiting for inference service zonemercy-viral-ss-v0-1e5ep2-v30 to be ready
Failed to get response for submission chaiml-elo-alignment-run-3_v36: ('http://chaiml-llama-8b-pairwis-8189-v27-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:34994->127.0.0.1:8080: read: connection reset by peer\n')
Inference service zonemercy-viral-ss-v0-1e5ep2-v30 ready after 130.48004388809204s
Pipeline stage MKMLDeployer completed in 131.32s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.84688138961792s
Received healthy response to inference request in 2.578669309616089s
Received healthy response to inference request in 2.4273316860198975s
Received healthy response to inference request in 2.2910714149475098s
Received healthy response to inference request in 2.5909502506256104s
5 requests
0 failed requests
5th percentile: 2.3183234691619874
10th percentile: 2.345575523376465
20th percentile: 2.40007963180542
30th percentile: 2.4575992107391356
40th percentile: 2.518134260177612
50th percentile: 2.578669309616089
60th percentile: 2.5835816860198975
70th percentile: 2.588494062423706
80th percentile: 2.642136478424072
90th percentile: 2.7445089340209963
95th percentile: 2.795695161819458
99th percentile: 2.8366441440582277
mean time: 2.5469808101654055
Pipeline stage StressChecker completed in 14.47s
Shutdown handler de-registered
zonemercy-viral-ss-v0-1e5ep2_v30 status is now deployed due to DeploymentManager action
zonemercy-viral-ss-v0-1e5ep2_v30 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7de62af7-22ad-45d7-8c4d-98745d96d11e, 1f414c04-d32c-441f-bec2-506f199e2332\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:47 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4af65e49-e488-4d50-85ee-9bdb928979df, 23d82554-0ddb-4bd7-aee4-9dfb0a00786f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'339004c5-68a7-4d55-888b-c62709469816, d98f9c86-5cd5-465e-828f-5b2eb8a55e67\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:37 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d1783f56-a66d-4c5c-902e-12cfc9a6f763, 312e791f-6975-4927-8746-43dac83c08c8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:38 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b2c0d04b-7392-4643-98c0-99a762960d95, 6c806c65-296d-4c28-ba5f-b00f24b48287\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:39:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'13914bb4-13f3-4676-af7a-276fe810e5c4, 4cc9d2ec-1abb-4ff4-bfe1-a1ff964cd2f4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'83f9f8a5-f03c-4255-af62-f1a9892e5647, 410c4714-2af5-4a01-9f14-70f1b3f1ceb4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'aae67c13-7fcb-4182-b689-f9ef9a10b7af, 72ad76ff-f789-4191-b143-5a7d2eea4835\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:48 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fe06eafb-b4ab-4e5d-b26e-95492cc0f35c, e42bb778-5aa0-4b62-beb4-f175fafa824b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:47 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fc20e3d6-0cbe-43b9-90f2-27a835f6982a, ce4673bf-e953-41b3-b3af-6021f474f532\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:02:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a12441b8-8bb6-46bf-98cf-2ff88a5873b8, dd098869-e0d0-4181-88d2-7842cb90946d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'823930e4-d4e8-4041-b76d-55481ab07e26, 5e0d9c3d-0cdd-4760-99d6-2fa87cb2f1b2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e4d27a25-c846-442c-9951-b1cdf5b8fb62, 2c4287e3-e756-4ec6-ad73-372ea3ea182b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:57 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'de500dd1-fcfe-4c3d-bfa9-2bd80da061a3, 259caf69-ded5-4a61-99d8-09a2ce8270f9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'08f054b4-acf9-4fe0-a60b-581263f61a77, 9834a533-d6c7-436d-9a79-22646f8fe37f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:57 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'eb20324a-0629-4874-90c4-36802cb932a3, b7731dcf-a14c-45e2-8e1b-488e30eeb401\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:32:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f381d2a7-d4e8-420f-ae50-c10c0ae4da2a, e7e1fa01-05f5-4403-bac0-1705a544edda\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:37:02 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'de9e6c6f-b112-439b-9b83-d0395b65decd, fe2f17bd-a595-4741-9231-b66e9316b463\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:42:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'afa81a33-61cb-43f2-b37c-af18ad11e0d8, 53edae95-d0a5-4b4f-a0ba-c081cc258dcd\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:47:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f3875a36-32f7-41c8-be13-b8a8ec61d89f, 549759fe-c55a-4d37-9821-e6b24214f696\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:52:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7b72a82b-f958-40b5-abce-1979ab8b9c8b, 41d34e92-5acf-480b-ae43-f1454641191d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:57:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ca7dbbbe-3389-4259-b31a-9cea33d35b4f, ec6dd956-c9ff-4279-a565-e3d2d0c279b4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:02:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2e68528-b1b3-4fc3-8a3c-f03332db184a, bc132d8c-fab7-4e38-8b58-cab19bbffc07\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:07:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a2b1f61d-4eda-4534-8ce8-ecb7205d7b2c, 10020c7e-aae5-43f1-b83c-7ec019d5d9c4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:12:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4fc36a30-ee16-4c51-9c3f-1bfb52da9379, 0282f303-261a-4a98-ae0c-23b1b7045c20\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:17:18 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8a3351fa-ccda-4244-ad56-c7a8b02a0ec4, 6fcb9e2e-3dde-4d07-8d5b-79b23f601860\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:22:22 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v30
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5ep2-v30 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9d63f492-08df-4d1c-bf7d-c2c73c6689ad, d76c7d46-f7c5-4970-9c80-f15bf05f9601\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:27:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
zonemercy-viral-ss-v0-1e5ep2_v30 status is now torndown due to DeploymentManager action