developer_uid: sao10k
submission_id: sao10k-12b-party-v1_v2
model_name: party-1-test
model_group: Sao10K/12b-party-v1
status: torndown
timestamp: 2024-09-28T09:00:47+00:00
num_battles: 4459
num_wins: 2320
celo_rating: 1268.91
family_friendly_score: 0.5965054385558899
family_friendly_standard_error: 0.007447150489039555
submission_type: basic
model_repo: Sao10K/12b-party-v1
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: party-1-test
ineligible_reason: num_battles<5000
is_internal_developer: False
language_model: Sao10K/12b-party-v1
model_size: 13B
ranking_group: single
us_pacific_date: 2024-09-28
win_ratio: 0.5202960305001121
generation_params: {'temperature': 0.75, 'top_p': 0.8, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n\n', '\nYou:', '<|im_end|>', '</s>', '[/INST]'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '[INST]system\n{memory} [/INST]\n', 'prompt_template': '[INST]user\n{prompt} [/INST]\n', 'bot_template': '[INST]model\n{bot_name}: {message} [/INST]\n', 'user_template': '[INST]user\n{user_name}: {message} [/INST]\n', 'response_template': '[INST]model\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name sao10k-12b-party-v1-v2-mkmlizer
Waiting for job on sao10k-12b-party-v1-v2-mkmlizer to finish
sao10k-12b-party-v1-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
sao10k-12b-party-v1-v2-mkmlizer: ║     _____            __           __                                ║
sao10k-12b-party-v1-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
sao10k-12b-party-v1-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
sao10k-12b-party-v1-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
sao10k-12b-party-v1-v2-mkmlizer: ║       /___/                                                         ║
sao10k-12b-party-v1-v2-mkmlizer: ║                                                                     ║
sao10k-12b-party-v1-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
sao10k-12b-party-v1-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
sao10k-12b-party-v1-v2-mkmlizer: ║  https://mk1.ai                                                     ║
sao10k-12b-party-v1-v2-mkmlizer: ║                                                                     ║
sao10k-12b-party-v1-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
sao10k-12b-party-v1-v2-mkmlizer: ║  belonging to:                                                      ║
sao10k-12b-party-v1-v2-mkmlizer: ║                                                                     ║
sao10k-12b-party-v1-v2-mkmlizer: ║  Chai Research Corp.                                                ║
sao10k-12b-party-v1-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
sao10k-12b-party-v1-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
sao10k-12b-party-v1-v2-mkmlizer: ║                                                                     ║
sao10k-12b-party-v1-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
sao10k-12b-party-v1-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
sao10k-12b-party-v1-v1-mkmlizer: ║     _____            __           __                                ║
sao10k-12b-party-v1-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
sao10k-12b-party-v1-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
sao10k-12b-party-v1-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
sao10k-12b-party-v1-v1-mkmlizer: ║       /___/                                                         ║
sao10k-12b-party-v1-v1-mkmlizer: ║                                                                     ║
sao10k-12b-party-v1-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
sao10k-12b-party-v1-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
sao10k-12b-party-v1-v1-mkmlizer: ║  https://mk1.ai                                                     ║
sao10k-12b-party-v1-v1-mkmlizer: ║                                                                     ║
sao10k-12b-party-v1-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
sao10k-12b-party-v1-v1-mkmlizer: ║  belonging to:                                                      ║
sao10k-12b-party-v1-v1-mkmlizer: ║                                                                     ║
sao10k-12b-party-v1-v1-mkmlizer: ║  Chai Research Corp.                                                ║
sao10k-12b-party-v1-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
sao10k-12b-party-v1-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
sao10k-12b-party-v1-v1-mkmlizer: ║                                                                     ║
sao10k-12b-party-v1-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
sao10k-12b-party-v1-v2-mkmlizer: Downloaded to shared memory in 50.313s
sao10k-12b-party-v1-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp6bupp2t2, device:0
sao10k-12b-party-v1-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
sao10k-12b-party-v1-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py:55: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
sao10k-12b-party-v1-v2-mkmlizer:   tensors = torch.load(model_shard_filename, map_location=torch.device(self.device), mmap=True)
sao10k-12b-party-v1-v1-mkmlizer: Downloaded to shared memory in 50.676s
sao10k-12b-party-v1-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpt9c_hheq, device:0
sao10k-12b-party-v1-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
sao10k-12b-party-v1-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py:55: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
sao10k-12b-party-v1-v1-mkmlizer:   tensors = torch.load(model_shard_filename, map_location=torch.device(self.device), mmap=True)
sao10k-12b-party-v1-v2-mkmlizer: quantized model in 37.431s
sao10k-12b-party-v1-v2-mkmlizer: Processed model Sao10K/12b-party-v1 in 87.745s
sao10k-12b-party-v1-v2-mkmlizer: creating bucket guanaco-mkml-models
sao10k-12b-party-v1-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
sao10k-12b-party-v1-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/sao10k-12b-party-v1-v2
sao10k-12b-party-v1-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/sao10k-12b-party-v1-v2/config.json
sao10k-12b-party-v1-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/sao10k-12b-party-v1-v2/special_tokens_map.json
sao10k-12b-party-v1-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/sao10k-12b-party-v1-v2/tokenizer_config.json
sao10k-12b-party-v1-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/sao10k-12b-party-v1-v2/tokenizer.json
sao10k-12b-party-v1-v1-mkmlizer: quantized model in 37.957s
sao10k-12b-party-v1-v1-mkmlizer: Processed model Sao10K/12b-party-v1 in 88.634s
sao10k-12b-party-v1-v1-mkmlizer: creating bucket guanaco-mkml-models
sao10k-12b-party-v1-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
sao10k-12b-party-v1-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/sao10k-12b-party-v1-v1
sao10k-12b-party-v1-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/sao10k-12b-party-v1-v1/config.json
sao10k-12b-party-v1-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/sao10k-12b-party-v1-v1/special_tokens_map.json
sao10k-12b-party-v1-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/sao10k-12b-party-v1-v1/tokenizer_config.json
sao10k-12b-party-v1-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/sao10k-12b-party-v1-v1/tokenizer.json
sao10k-12b-party-v1-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/sao10k-12b-party-v1-v1/flywheel_model.0.safetensors
sao10k-12b-party-v1-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/363 [00:00<00:09, 36.60it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 53.50it/s]
Loading 0:   6%|▌         | 22/363 [00:00<00:05, 57.57it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:05, 59.42it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:05, 59.17it/s]
Loading 0:  13%|█▎        | 49/363 [00:00<00:05, 61.01it/s]
Loading 0:  16%|█▌        | 58/363 [00:00<00:04, 64.64it/s]
Loading 0:  18%|█▊        | 65/363 [00:02<00:16, 17.97it/s]
Loading 0:  19%|█▉        | 70/363 [00:02<00:14, 19.80it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:10, 26.22it/s]
Loading 0:  24%|██▍       | 88/363 [00:02<00:08, 32.95it/s]
Loading 0:  27%|██▋       | 97/363 [00:02<00:06, 38.27it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:05, 43.45it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 48.60it/s]
Loading 0:  34%|███▍      | 124/363 [00:03<00:04, 51.93it/s]
Loading 0:  37%|███▋      | 133/363 [00:03<00:03, 57.55it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:11, 19.58it/s]
Loading 0:  42%|████▏     | 151/363 [00:04<00:08, 24.66it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 30.14it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 35.70it/s]
Loading 0:  49%|████▉     | 178/363 [00:04<00:04, 40.56it/s]
Loading 0:  52%|█████▏    | 187/363 [00:05<00:03, 45.56it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:03, 48.50it/s]
Loading 0:  56%|█████▋    | 205/363 [00:05<00:03, 50.42it/s]
Loading 0:  59%|█████▉    | 214/363 [00:05<00:02, 56.38it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:07, 19.64it/s]
Loading 0:  64%|██████▍   | 232/363 [00:06<00:05, 24.56it/s]
Loading 0:  66%|██████▋   | 241/363 [00:07<00:04, 30.15it/s]
Loading 0:  69%|██████▉   | 250/363 [00:07<00:03, 35.03it/s]
Loading 0:  71%|███████▏  | 259/363 [00:07<00:02, 39.97it/s]
Loading 0:  74%|███████▍  | 268/363 [00:07<00:02, 44.71it/s]
Loading 0:  76%|███████▋  | 277/363 [00:07<00:01, 48.02it/s]
Loading 0:  79%|███████▉  | 286/363 [00:07<00:01, 53.82it/s]
Loading 0:  81%|████████▏ | 295/363 [00:07<00:01, 60.25it/s]
Loading 0:  84%|████████▎ | 304/363 [00:09<00:02, 19.96it/s]
Loading 0:  86%|████████▌ | 313/363 [00:09<00:02, 24.92it/s]
Loading 0:  89%|████████▊ | 322/363 [00:09<00:01, 30.32it/s]
Loading 0:  91%|█████████ | 331/363 [00:09<00:00, 34.80it/s]
Loading 0:  94%|█████████▎| 340/363 [00:09<00:00, 41.90it/s]
Loading 0:  96%|█████████▌| 349/363 [00:09<00:00, 48.50it/s]
Loading 0:  99%|█████████▊| 358/363 [00:09<00:00, 55.20it/s]
                                                            

Job sao10k-12b-party-v1-v1-mkmlizer completed after 215.5s with status: succeeded
Stopping job with name sao10k-12b-party-v1-v1-mkmlizer
Pipeline stage MKMLizer completed in 216.88s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service sao10k-12b-party-v1-v1
Waiting for inference service sao10k-12b-party-v1-v1 to be ready
sao10k-12b-party-v1-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/sao10k-12b-party-v1-v2/flywheel_model.0.safetensors
sao10k-12b-party-v1-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/363 [00:00<00:09, 38.11it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:05, 62.08it/s]
Loading 0:   6%|▌         | 22/363 [00:00<00:05, 66.58it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:04, 68.87it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:04, 70.67it/s]
Loading 0:  13%|█▎        | 49/363 [00:00<00:04, 67.85it/s]
Loading 0:  16%|█▌        | 58/363 [00:00<00:04, 65.94it/s]
Loading 0:  18%|█▊        | 65/363 [00:02<00:16, 18.15it/s]
Loading 0:  19%|█▉        | 70/363 [00:02<00:14, 20.82it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:10, 28.14it/s]
Loading 0:  24%|██▍       | 88/363 [00:02<00:07, 34.90it/s]
Loading 0:  27%|██▋       | 97/363 [00:02<00:06, 41.50it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:05, 46.58it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 49.08it/s]
Loading 0:  34%|███▍      | 124/363 [00:02<00:04, 52.13it/s]
Loading 0:  37%|███▋      | 133/363 [00:03<00:04, 55.29it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:11, 19.24it/s]
Loading 0:  42%|████▏     | 151/363 [00:04<00:08, 24.75it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 31.05it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 36.26it/s]
Loading 0:  49%|████▉     | 178/363 [00:04<00:04, 41.88it/s]
Loading 0:  52%|█████▏    | 187/363 [00:04<00:03, 46.42it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:03, 48.78it/s]
Loading 0:  56%|█████▋    | 205/363 [00:05<00:03, 52.37it/s]
Loading 0:  59%|█████▉    | 214/363 [00:05<00:02, 52.81it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:07, 18.98it/s]
Loading 0:  64%|██████▍   | 232/363 [00:06<00:05, 24.40it/s]
Loading 0:  66%|██████▋   | 241/363 [00:06<00:04, 30.42it/s]
Loading 0:  69%|██████▉   | 250/363 [00:06<00:03, 36.62it/s]
Loading 0:  71%|███████▏  | 259/363 [00:07<00:02, 42.19it/s]
Loading 0:  74%|███████▍  | 268/363 [00:07<00:02, 47.12it/s]
Loading 0:  76%|███████▋  | 277/363 [00:07<00:01, 51.64it/s]
Loading 0:  79%|███████▉  | 286/363 [00:07<00:01, 56.42it/s]
Loading 0:  81%|████████▏ | 295/363 [00:07<00:01, 58.79it/s]
Loading 0:  84%|████████▎ | 304/363 [00:08<00:02, 19.92it/s]
Loading 0:  86%|████████▌ | 313/363 [00:08<00:01, 25.44it/s]
Loading 0:  89%|████████▊ | 322/363 [00:09<00:01, 31.89it/s]
Loading 0:  91%|█████████ | 331/363 [00:09<00:00, 37.81it/s]
Loading 0:  94%|█████████▎| 340/363 [00:09<00:00, 43.76it/s]
Loading 0:  96%|█████████▌| 349/363 [00:09<00:00, 49.65it/s]
Loading 0:  99%|█████████▊| 358/363 [00:09<00:00, 53.79it/s]
                                                            

Job sao10k-12b-party-v1-v2-mkmlizer completed after 185.12s with status: succeeded
Stopping job with name sao10k-12b-party-v1-v2-mkmlizer
Pipeline stage MKMLizer completed in 185.54s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service sao10k-12b-party-v1-v2
Waiting for inference service sao10k-12b-party-v1-v2 to be ready
Inference service sao10k-12b-party-v1-v1 ready after 240.55661940574646s
Pipeline stage MKMLDeployer completed in 241.06s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3291003704071045s
Received healthy response to inference request in 1.7127695083618164s
Received healthy response to inference request in 1.6380832195281982s
Received healthy response to inference request in 1.679422378540039s
Inference service sao10k-12b-party-v1-v2 ready after 240.54252409934998s
Pipeline stage MKMLDeployer completed in 240.94s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9002037048339844s
5 requests
0 failed requests
5th percentile: 1.6463510513305664
10th percentile: 1.6546188831329345
20th percentile: 1.671154546737671
30th percentile: 1.6860918045043944
40th percentile: 1.6994306564331054
50th percentile: 1.7127695083618164
60th percentile: 1.7877431869506837
70th percentile: 1.8627168655395507
80th percentile: 1.9859830379486085
90th percentile: 2.1575417041778566
95th percentile: 2.2433210372924806
99th percentile: 2.3119445037841797
mean time: 1.8519158363342285
Pipeline stage StressChecker completed in 10.79s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.2137086391448975s
Received healthy response to inference request in 1.7253930568695068s
Received healthy response to inference request in 1.4227051734924316s
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.08s
Shutdown handler de-registered
sao10k-12b-party-v1_v1 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 1.9339110851287842s
Received healthy response to inference request in 1.8203401565551758s
5 requests
0 failed requests
5th percentile: 1.4832427501678467
10th percentile: 1.5437803268432617
20th percentile: 1.6648554801940918
30th percentile: 1.7443824768066407
40th percentile: 1.7823613166809082
50th percentile: 1.8203401565551758
60th percentile: 1.8657685279846192
70th percentile: 1.9111968994140625
80th percentile: 1.9898705959320069
90th percentile: 2.101789617538452
95th percentile: 2.1577491283416745
99th percentile: 2.202516736984253
mean time: 1.8232116222381591
Pipeline stage StressChecker completed in 10.07s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.93s
Shutdown handler de-registered
sao10k-12b-party-v1_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service sao10k-12b-party-v1-v2-profiler
Waiting for inference service sao10k-12b-party-v1-v2-profiler to be ready
Tearing down inference service sao10k-12b-party-v1-v2-profiler
%s, retrying in %s seconds...
Creating inference service sao10k-12b-party-v1-v2-profiler
Waiting for inference service sao10k-12b-party-v1-v2-profiler to be ready
Tearing down inference service sao10k-12b-party-v1-v2-profiler
%s, retrying in %s seconds...
Creating inference service sao10k-12b-party-v1-v2-profiler
Waiting for inference service sao10k-12b-party-v1-v2-profiler to be ready
Tearing down inference service sao10k-12b-party-v1-v2-profiler
clean up pipeline due to error=%s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.16s
Shutdown handler de-registered
sao10k-12b-party-v1_v2 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'23d2fedc-1506-40d8-a83d-c12ed8001cde, dc2e3bd1-7011-4cbc-bbf9-be01119f7da8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'15b1c2cf-a30d-477a-b746-0f220542288a, f7d68e27-e8c7-408e-aabc-cc99b6750180\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3d30a8ee-445b-4954-a06c-a7fe9db20efb, 29880e70-817b-43d8-874c-3a6f705ce2fc\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f4407507-d8b3-414d-b5fe-3a413d334a1f, 152e6d27-4dbe-4cef-97fb-83a771f928aa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'558270aa-53ef-49f4-8e98-fca686e38151, 04f5466e-5a74-4559-997b-c29f1225c844\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:37:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c0b86414-564c-42eb-96ad-555694c66034, 811bfc61-dd85-46a7-8ad9-2912cb75508b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'59d76baa-d3c4-4f2c-a708-d9a17f74b4dd, c9825dd7-a015-461b-94e8-9ad780d59f5b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'82ad5af8-8f99-49b7-aa1f-0c80879fe0a6, c62500e2-9b88-45d5-8730-7e4f47aa02c2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d4a14590-4e2b-450e-8d86-bc5cb4eca029, faf5f501-a8d8-4777-9917-ca043ef4bfec\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5b9b49ce-14cc-4780-baa3-7c29a59262f2, 46e1b454-f5ef-4c0b-8029-638cf745b232\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0fe54ecf-25e7-4f04-9c2a-71a9a4128b25, e47c3d6d-d5c0-466e-b65b-072e4bb7410d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'84c5afe6-d5b2-4a0f-889f-636d6cc7ddff, 4bb48d32-bf57-4b82-af18-a4e9ae72bc83\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8bd59a6c-6cd2-4b92-ab8a-e296b05ca0d8, 2725954c-b083-4190-853a-04a141adcc77\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6f484ca9-3fa6-4143-9cad-258921194334, 7db89913-8cbc-4ac5-9444-73241cde6bd0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c8327fec-0278-4d17-bd2a-3bdfc8755771, 34843b79-948e-4044-9950-9899dd11f1d4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'45ee9d44-5cf0-473f-b540-47061b2a1ee6, 528411dd-8e28-4b88-9d3f-3591209f6f26\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:34 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'22d7142e-2caa-44d9-82f8-61a7a36b0cd2, 68cd7be2-76fc-4020-b31e-352a9dddc619\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7cf200ae-e564-457c-b583-ca0dcc0b041f, d1cb39bd-5e9c-42ea-bf4c-8f00bbc53873\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8e676a32-d839-4d4a-ae12-c85a8e0df9ea, 9c1201d5-04f0-471c-9ff6-4963b704fcb5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5302c4f0-249b-45cc-90e9-464c8d167d9b, 9cb63258-40eb-4660-b97c-81fc5b52ba27\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'07fb6742-d7e6-40d5-b6fc-6118e655846c, d9dc0421-f3fa-4f98-9fde-3de069cb911a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:41 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5a79fd5f-8b86-4b2e-b8d2-1c36daaff8be, 66b797b3-d489-4bc0-8a0b-468d8b439e2f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2c209454-fa9b-4216-b2d7-0b5538ad5e6b, 96888fbb-78c9-46cc-a016-127066a92f19\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'776cc667-4780-40d8-b1b0-8cae1b31ee4e, 49dbf773-b04a-408d-8270-e9fc65a30f6e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'05d78364-b8b8-416d-8d0f-2290c5acb454, 6bb3524e-840b-4829-bb64-7296021eff8a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f5230487-a980-4c7a-928c-a4d47d798f2e, c3c2227c-b5e8-41c8-add2-96af80386750\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:47 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of sao10k-12b-party-v1_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
%s, retrying in %s seconds...
Checking if service sao10k-12b-party-v1-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'54c0928f-dd1c-4a87-bb4a-e875a91d2744, b1f71f5f-0643-4326-9e8d-5efab1aec69c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
sao10k-12b-party-v1_v2 status is now torndown due to DeploymentManager action