jic062-nemo-v1-9

developer_uid: chace9580
submission_id: jic062-nemo-v1-9_v7
model_name: jic062-nemo-v1-9_v7
model_group: jic062/Nemo-v1.9
status: torndown
timestamp: 2024-09-30T05:30:08+00:00
num_battles: 112525
num_wins: 56094
celo_rating: 1255.82
family_friendly_score: 0.5887515810344238
family_friendly_standard_error: 0.004945745793980714
submission_type: basic
model_repo: jic062/Nemo-v1.9
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: jic062-nemo-v1-9_v7
is_internal_developer: False
language_model: jic062/Nemo-v1.9
model_size: 13B
ranking_group: single
us_pacific_date: 2024-09-29
win_ratio: 0.4985025549877805
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '/s', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-nemo-v1-9-v7-mkmlizer
Waiting for job on jic062-nemo-v1-9-v7-mkmlizer to finish
jic062-nemo-v1-9-v7-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-nemo-v1-9-v7-mkmlizer: ║     _____            __           __                                ║
jic062-nemo-v1-9-v7-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
jic062-nemo-v1-9-v7-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
jic062-nemo-v1-9-v7-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
jic062-nemo-v1-9-v7-mkmlizer: ║       /___/                                                         ║
jic062-nemo-v1-9-v7-mkmlizer: ║                                                                     ║
jic062-nemo-v1-9-v7-mkmlizer: ║  Version: 0.11.12                                                   ║
jic062-nemo-v1-9-v7-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
jic062-nemo-v1-9-v7-mkmlizer: ║  https://mk1.ai                                                     ║
jic062-nemo-v1-9-v7-mkmlizer: ║                                                                     ║
jic062-nemo-v1-9-v7-mkmlizer: ║  The license key for the current software has been verified as      ║
jic062-nemo-v1-9-v7-mkmlizer: ║  belonging to:                                                      ║
jic062-nemo-v1-9-v7-mkmlizer: ║                                                                     ║
jic062-nemo-v1-9-v7-mkmlizer: ║  Chai Research Corp.                                                ║
jic062-nemo-v1-9-v7-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
jic062-nemo-v1-9-v7-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
jic062-nemo-v1-9-v7-mkmlizer: ║                                                                     ║
jic062-nemo-v1-9-v7-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
jic062-nemo-v1-9-v6-mkmlizer: Downloaded to shared memory in 29.740s
jic062-nemo-v1-9-v6-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpndnc8apl, device:0
jic062-nemo-v1-9-v6-mkmlizer: Saving flywheel model at /dev/shm/model_cache
jic062-nemo-v1-9-v7-mkmlizer: Downloaded to shared memory in 27.978s
jic062-nemo-v1-9-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmptt0_7s0b, device:0
jic062-nemo-v1-9-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
jic062-nemo-v1-9-v6-mkmlizer: quantized model in 35.033s
jic062-nemo-v1-9-v6-mkmlizer: Processed model jic062/Nemo-v1.9 in 64.774s
jic062-nemo-v1-9-v6-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-nemo-v1-9-v6-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-nemo-v1-9-v6
jic062-nemo-v1-9-v6-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-nemo-v1-9-v6/config.json
jic062-nemo-v1-9-v6-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-nemo-v1-9-v6/special_tokens_map.json
jic062-nemo-v1-9-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-nemo-v1-9-v6/tokenizer_config.json
jic062-nemo-v1-9-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-nemo-v1-9-v6/tokenizer.json
jic062-nemo-v1-9-v6-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/jic062-nemo-v1-9-v6/flywheel_model.0.safetensors
jic062-nemo-v1-9-v6-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.56it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 51.70it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 46.69it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 45.52it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 51.45it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 47.65it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:06, 46.20it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 51.54it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 47.94it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 36.54it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:07, 37.65it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:06, 41.81it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:06, 41.16it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:06, 41.50it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:05, 45.97it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:05, 45.61it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:05, 45.95it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:05, 43.28it/s]
Loading 0:  30%|███       | 110/363 [00:02<00:05, 44.74it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 45.40it/s]
Loading 0:  33%|███▎      | 120/363 [00:02<00:05, 43.38it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:05, 45.43it/s]
Loading 0:  36%|███▋      | 132/363 [00:02<00:05, 42.85it/s]
Loading 0:  38%|███▊      | 137/363 [00:03<00:05, 42.09it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:06, 32.89it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 34.38it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 33.56it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 38.83it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:05, 40.17it/s]
Loading 0:  46%|████▌     | 166/363 [00:03<00:04, 41.64it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:04, 40.69it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 40.11it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:03, 45.30it/s]
Loading 0:  52%|█████▏    | 190/363 [00:04<00:03, 43.36it/s]
Loading 0:  54%|█████▎    | 195/363 [00:04<00:03, 42.33it/s]
Loading 0:  55%|█████▌    | 201/363 [00:04<00:03, 46.49it/s]
Loading 0:  57%|█████▋    | 206/363 [00:04<00:03, 46.35it/s]
Loading 0:  58%|█████▊    | 211/363 [00:04<00:03, 46.12it/s]
Loading 0:  60%|█████▉    | 217/363 [00:05<00:03, 43.48it/s]
Loading 0:  61%|██████    | 222/363 [00:05<00:03, 44.87it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:04, 31.31it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:04, 31.50it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 37.09it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:03, 39.25it/s]
Loading 0:  68%|██████▊   | 247/363 [00:05<00:02, 40.74it/s]
Loading 0:  70%|██████▉   | 253/363 [00:06<00:02, 40.37it/s]
Loading 0:  71%|███████   | 258/363 [00:06<00:02, 40.41it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 44.77it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 44.82it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 44.48it/s]
Loading 0:  77%|███████▋  | 279/363 [00:06<00:01, 45.50it/s]
Loading 0:  78%|███████▊  | 284/363 [00:06<00:02, 37.48it/s]
Loading 0:  80%|████████  | 291/363 [00:06<00:01, 45.07it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 45.15it/s]
Loading 0:  83%|████████▎ | 301/363 [00:07<00:01, 46.28it/s]
Loading 0:  84%|████████▍ | 306/363 [00:13<00:22,  2.50it/s]
Loading 0:  85%|████████▌ | 310/363 [00:13<00:16,  3.25it/s]
Loading 0:  87%|████████▋ | 315/363 [00:14<00:10,  4.55it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:06,  6.28it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:04,  8.85it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:02, 11.49it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 16.44it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 20.18it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 23.55it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 30.22it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 32.78it/s]
                                                            
Job jic062-nemo-v1-9-v6-mkmlizer completed after 92.94s with status: succeeded
Stopping job with name jic062-nemo-v1-9-v6-mkmlizer
Pipeline stage MKMLizer completed in 93.72s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service jic062-nemo-v1-9-v6
Waiting for inference service jic062-nemo-v1-9-v6 to be ready
jic062-nemo-v1-9-v7-mkmlizer: quantized model in 35.140s
jic062-nemo-v1-9-v7-mkmlizer: Processed model jic062/Nemo-v1.9 in 63.118s
jic062-nemo-v1-9-v7-mkmlizer: creating bucket guanaco-mkml-models
jic062-nemo-v1-9-v7-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-nemo-v1-9-v7-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-nemo-v1-9-v7
jic062-nemo-v1-9-v7-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-nemo-v1-9-v7/special_tokens_map.json
jic062-nemo-v1-9-v7-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-nemo-v1-9-v7/config.json
jic062-nemo-v1-9-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-nemo-v1-9-v7/tokenizer_config.json
jic062-nemo-v1-9-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-nemo-v1-9-v7/tokenizer.json
jic062-nemo-v1-9-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/jic062-nemo-v1-9-v7/flywheel_model.0.safetensors
jic062-nemo-v1-9-v7-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.19it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 51.86it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 47.83it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:06, 49.49it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 52.16it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 49.94it/s]
Loading 0:  12%|█▏        | 43/363 [00:00<00:06, 50.83it/s]
Loading 0:  13%|█▎        | 49/363 [00:00<00:05, 52.98it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 50.78it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 37.53it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:07, 37.87it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:06, 42.48it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:06, 42.99it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:06, 42.92it/s]
Loading 0:  25%|██▍       | 90/363 [00:01<00:05, 47.52it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:05, 46.00it/s]
Loading 0:  28%|██▊       | 101/363 [00:02<00:05, 45.30it/s]
Loading 0:  30%|██▉       | 108/363 [00:02<00:04, 51.09it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:05, 45.71it/s]
Loading 0:  33%|███▎      | 119/363 [00:02<00:05, 44.57it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:04, 48.42it/s]
Loading 0:  36%|███▋      | 132/363 [00:02<00:04, 46.74it/s]
Loading 0:  38%|███▊      | 137/363 [00:02<00:04, 45.44it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:06, 35.11it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 35.32it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 34.51it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:05, 40.25it/s]
Loading 0:  45%|████▍     | 163/363 [00:03<00:04, 41.17it/s]
Loading 0:  46%|████▋     | 168/363 [00:03<00:04, 41.05it/s]
Loading 0:  48%|████▊     | 175/363 [00:03<00:04, 46.30it/s]
Loading 0:  50%|████▉     | 181/363 [00:04<00:03, 45.71it/s]
Loading 0:  51%|█████     | 186/363 [00:04<00:03, 44.47it/s]
Loading 0:  53%|█████▎    | 192/363 [00:04<00:03, 47.91it/s]
Loading 0:  54%|█████▍    | 197/363 [00:04<00:03, 46.30it/s]
Loading 0:  56%|█████▌    | 202/363 [00:04<00:03, 46.79it/s]
Loading 0:  57%|█████▋    | 207/363 [00:04<00:03, 47.65it/s]
Loading 0:  58%|█████▊    | 212/363 [00:04<00:03, 40.14it/s]
Loading 0:  61%|██████    | 220/363 [00:04<00:02, 49.71it/s]
Loading 0:  62%|██████▏   | 226/363 [00:05<00:04, 32.54it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:03, 34.04it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 39.26it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:03, 40.32it/s]
Loading 0:  68%|██████▊   | 247/363 [00:05<00:02, 41.56it/s]
Loading 0:  70%|██████▉   | 253/363 [00:05<00:02, 41.54it/s]
Loading 0:  71%|███████   | 258/363 [00:05<00:02, 40.69it/s]
Loading 0:  73%|███████▎  | 265/363 [00:06<00:02, 45.65it/s]
Loading 0:  75%|███████▍  | 271/363 [00:06<00:02, 43.69it/s]
Loading 0:  76%|███████▌  | 276/363 [00:06<00:02, 43.28it/s]
Loading 0:  78%|███████▊  | 283/363 [00:06<00:01, 47.90it/s]
Loading 0:  80%|███████▉  | 289/363 [00:06<00:01, 45.98it/s]
Loading 0:  81%|████████  | 294/363 [00:06<00:01, 43.53it/s]
Loading 0:  83%|████████▎ | 300/363 [00:06<00:01, 47.50it/s]
Loading 0:  84%|████████▍ | 305/363 [00:13<00:21,  2.66it/s]
Loading 0:  85%|████████▌ | 309/363 [00:13<00:15,  3.39it/s]
Loading 0:  86%|████████▌ | 313/363 [00:13<00:11,  4.39it/s]
Loading 0:  88%|████████▊ | 320/363 [00:13<00:06,  6.86it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:03,  9.44it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:02, 12.07it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 17.03it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 21.01it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 24.23it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 31.07it/s]
Loading 0: 100%|█████████▉| 362/363 [00:14<00:00, 34.32it/s]
                                                            
Job jic062-nemo-v1-9-v7-mkmlizer completed after 92.89s with status: succeeded
Stopping job with name jic062-nemo-v1-9-v7-mkmlizer
Pipeline stage MKMLizer completed in 93.25s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service jic062-nemo-v1-9-v7
Waiting for inference service jic062-nemo-v1-9-v7 to be ready
Inference service jic062-nemo-v1-9-v6 ready after 140.32259440422058s
Pipeline stage MKMLDeployer completed in 140.74s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service jic062-nemo-v1-9-v7 ready after 130.2933325767517s
Pipeline stage MKMLDeployer completed in 130.65s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1905291080474854s
Received healthy response to inference request in 1.71852445602417s
Received healthy response to inference request in 2.176602840423584s
Received healthy response to inference request in 1.3479950428009033s
Received healthy response to inference request in 2.0350611209869385s
Received healthy response to inference request in 1.2450690269470215s
Received healthy response to inference request in 1.8453304767608643s
Received healthy response to inference request in 2.001840114593506s
Received healthy response to inference request in 2.12471079826355s
5 requests
0 failed requests
5th percentile: 1.7438856601715087
10th percentile: 1.7692468643188477
20th percentile: 1.8199692726135255
30th percentile: 1.883276605606079
40th percentile: 1.9591688632965087
50th percentile: 2.0350611209869385
Received healthy response to inference request in 1.7497944831848145s
60th percentile: 2.070920991897583
5 requests
70th percentile: 2.1067808628082276
0 failed requests
80th percentile: 2.137874460220337
5th percentile: 1.2656542301177978
90th percentile: 2.164201784133911
10th percentile: 1.2862394332885743
95th percentile: 2.177365446090698
20th percentile: 1.327409839630127
99th percentile: 2.187896375656128
30th percentile: 1.4283549308776855
mean time: 1.9828311920166015
40th percentile: 1.58907470703125
50th percentile: 1.7497944831848145
Pipeline stage StressChecker completed in 11.60s
60th percentile: 1.850612735748291
run pipeline stage %s
70th percentile: 1.9514309883117675
Running pipeline stage TriggerMKMLProfilingPipeline
80th percentile: 2.0367926597595214
run_pipeline:run_in_cloud %s
90th percentile: 2.106697750091553
starting trigger_guanaco_pipeline args=%s
95th percentile: 2.1416502952575684
99th percentile: 2.169612331390381
mean time: 1.7042603015899658
Pipeline stage StressChecker completed in 10.25s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.93s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.39s
Shutdown handler de-registered
Shutdown handler de-registered
jic062-nemo-v1-9_v6 status is now deployed due to DeploymentManager action
jic062-nemo-v1-9_v7 status is now deployed due to DeploymentManager action
jic062-nemo-v1-9_v7 status is now inactive due to admin request
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4be5f89c-28cd-4ae8-871d-1731a88a7777, b2360d86-7dd9-4e79-bc01-e87dcea69db1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:07 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2f8ce0a-561e-4323-816d-4bc32b2c94e6, 6d634e96-1a73-47ba-a325-3e234c7697ad\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:52 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'65906361-9451-4904-888d-c638ef994c1c, 36f24577-691f-44c2-b775-000f35c1090d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:54 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cbac62ef-06aa-499f-9543-df64add2fe1e, 2ba58b5c-b174-4c9c-a8e9-a102a6a2d30e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a6ab9156-9153-443c-acad-46fec7c27654, 2b658657-f16f-4f9e-871c-7d11532cc930\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:35:57 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0354ab7c-6b7c-45fa-8db1-7d09567b9d41, 74b72566-2f51-4510-8ad0-cc3ae2ad7aa4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:40:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a69544d7-9877-4be7-8614-d51931d6d3f0, e1fa0a0b-db30-49ed-82ca-68b58d47a1aa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:45:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'685fc5f9-3f6f-4b8d-b90a-928a80018b71, 7c16af48-2839-4ed5-9a84-f26c9475a0a7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:50:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'835525b2-9d2d-4b73-b38e-356c3d9251aa, 6bd65db0-2850-4ce1-b274-323f060ecf1d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:55:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'343d4512-47fd-404d-81f3-ae8a7ef2837c, 5f5c2bd9-3146-4f50-912e-311e7ff431f1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'81615926-acff-4625-8606-8de78912eef4, 7075818a-69d6-42e6-94df-ce4ddaea9a2f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'28cf7cda-11de-48f7-a1cb-61a88b0b6b6c, 3309ac56-afe3-444f-a5c4-ec73406b6257\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3966ba46-9513-4217-921f-3a3239a9fb4e, a70897a2-ca2f-4b21-a1d7-20838796242b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ea6a6f9a-a4bb-4f5e-b584-c1b4075e62d7, 73999d99-92fa-4582-bc22-38e4121461e8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6804725c-a3a2-4c4c-8b54-5ec0357af04e, 971a8c3a-f318-4bb0-b8b6-c5984b1a4712\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6234d013-a859-49d2-a5f7-3a4b41c82ab4, c1733009-51ec-4736-a369-f1f44095a2f8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
Checking if service jic062-nemo-v1-8-v1 is running
admin requested tearing down of jic062-nemo-v1-9_v7
%s, retrying in %s seconds...
Shutdown handler not registered because Python interpreter is not running in the main thread
Checking if service jic062-nemo-v1-8-v1 is running
run pipeline %s
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a93ba586-5034-4c77-9475-bccf2adab235, c9cc2df8-0435-44cb-8cae-829a48135ff0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
run pipeline stage %s
Shutdown handler de-registered
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'10f8e757-36d4-4976-a803-30d2cdb324f1, c86ec98b-505c-4a80-8415-0eb1b24136ed\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:07 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'00f2cbb0-28a7-47e7-b0b3-87bcbbbcb20e, 9ad5a92a-690b-4afe-bcc5-c6cbeb81a0a3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'70829d12-3dc3-4091-97ba-6fb396c21d48, 517903bc-c101-4830-a526-296c7b9d0c1b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'39a01d8d-ca2a-42d6-a9f6-e6743991ac89, 22831292-a6a8-48c8-9d82-180f4691b451\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c06b3133-6643-4128-beae-08481f5eb626, 7a872923-ce8a-4088-9ee4-7129673de2c1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2c3a847a-84f8-45b8-8861-b6a41c0a2f4c, b99c194c-f661-4645-bda2-89a3c54a7e8d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dbaacb64-dee6-424f-a84b-d2c9a6bd06fe, 406dd98f-601c-4025-8981-f11a5d66297e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'83b9c04d-9c12-4da0-a4a1-8d8ce64a4544, 2751fda5-775c-4230-8b31-5a9581e1cace\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b76ba1ce-d2e8-4eeb-b1af-363403ad4bea, 16504c22-499c-4a3b-95de-83b72ee3cb8f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'514e0663-8e52-4e0d-9415-582a5ed543f7, 29aad4ab-6e1a-40e6-89a2-96f5c815f3da\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-9_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-9-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e22ecab8-23fa-45c7-b6a3-c52dad9e6607, c7a5a8db-c5f3-4220-aa4c-217eb24eb101\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
jic062-nemo-v1-9_v7 status is now torndown due to DeploymentManager action