Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name pawankrd-cosmosrp-12b-v16-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v16-mkmlizer to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
pawankrd-cosmosrp-12b-v16-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ _____ __ __ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ /___/ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Version: 0.11.12 ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
pawankrd-cosmosrp-12b-v16-mkmlizer: Traceback (most recent call last):
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 513, in http_get
pawankrd-cosmosrp-12b-v16-mkmlizer: hf_transfer.download(
pawankrd-cosmosrp-12b-v16-mkmlizer: Exception: Failed too many failures in parallel (3): PyErr { type: <class 'Exception'>, value: Exception('Error while downloading: reqwest::Error { kind: Status(503), url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("cdn-lfs-us-1.hf.co")), port: None, path: "/repos/f1/91/f410ead474dba039e33e8/259ee967fb0964649822a00d356589725e9c7158229a5ab597221e016190052c", query: Some("response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00004-of-00005.safetensors%3B+filename%3D%22model-00004-of-00005.safetensors%22%3B&Expires=1728481120&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyODQ4MTEyMH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zL2YxLzkxL2YxOTEyMzI3MDMwM2QyMTdhYWI1NDUyMzA2ZTYyNTM1MDkyNDA3YjIxZmFmNDEwZWFkNDc0ZGJhMDM5ZTMzZTgvMjU5ZWU5NjdmYjA5NjQ2NDk4MjJhMDBkMzU2NTg5NzI1ZTljNzE1ODIyOWE1YWI1OTcyMjFlMDE2MTkwMDUyYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=OyKKXHlReSD6RtsJAG0oCb20eeHKHGVyRWuM59tofn142Nnmt7%7E0oTCFhWsfMbKkeAjz5Gs2uGrjqlvUZ5Rg4n6aEk3vpKwBX%7E%7EbjTJ90W6WBFSqBtS1bFNlGlot1E0hffpms0ZqhTHqii-Jl4k8ahPLG-M%7E1E-MiWO36pNzbIqs6RB3PeGI2spH0lXW4gHwkBsyrvU7XZKvgLDuEWFmwmnJW20YCg0VMqly4aWF%7Exmn449dLFeXLgmc7-Xfd2My-L0k0AQMZyv%7E4%7EgboeJyw56Ojln79NjVTso%7Eh-yPzG3AMw7mwW%7EcVwlV76gZlNfjwhyBXBJRSC5l8xKt0ZyakQ__&Key-Pair-Id=K24J24Z295AEI9"), fragment: None } }'), traceback: None } (NoPermits)
pawankrd-cosmosrp-12b-v16-mkmlizer: The above exception was the direct cause of the following exception:
pawankrd-cosmosrp-12b-v16-mkmlizer: Traceback (most recent call last):
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/code/uploading/mkmlize.py", line 151, in <module>
pawankrd-cosmosrp-12b-v16-mkmlizer: cli()
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
pawankrd-cosmosrp-12b-v16-mkmlizer: return self.main(*args, **kwargs)
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
pawankrd-cosmosrp-12b-v16-mkmlizer: rv = self.invoke(ctx)
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
pawankrd-cosmosrp-12b-v16-mkmlizer: return _process_result(sub_ctx.command.invoke(sub_ctx))
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
pawankrd-cosmosrp-12b-v16-mkmlizer: return ctx.invoke(self.callback, **ctx.params)
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
pawankrd-cosmosrp-12b-v16-mkmlizer: return __callback(*args, **kwargs)
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/code/uploading/mkmlize.py", line 38, in quantize
pawankrd-cosmosrp-12b-v16-mkmlizer: temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/code/uploading/mkmlize.py", line 65, in download_to_shared_memory
pawankrd-cosmosrp-12b-v16-mkmlizer: snapshot_download(
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
pawankrd-cosmosrp-12b-v16-mkmlizer: return fn(*args, **kwargs)
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 292, in snapshot_download
pawankrd-cosmosrp-12b-v16-mkmlizer: _inner_hf_hub_download(file)
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 268, in _inner_hf_hub_download
pawankrd-cosmosrp-12b-v16-mkmlizer: return hf_hub_download(
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
pawankrd-cosmosrp-12b-v16-mkmlizer: return fn(*args, **kwargs)
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1202, in hf_hub_download
pawankrd-cosmosrp-12b-v16-mkmlizer: return _hf_hub_download_to_local_dir(
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1487, in _hf_hub_download_to_local_dir
pawankrd-cosmosrp-12b-v16-mkmlizer: _download_to_tmp_and_move(
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1884, in _download_to_tmp_and_move
pawankrd-cosmosrp-12b-v16-mkmlizer: http_get(
pawankrd-cosmosrp-12b-v16-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 524, in http_get
pawankrd-cosmosrp-12b-v16-mkmlizer: raise RuntimeError(
pawankrd-cosmosrp-12b-v16-mkmlizer: RuntimeError: An error occurred while downloading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.
Job pawankrd-cosmosrp-12b-v16-mkmlizer completed after 54.86s with status: failed
Stopping job with name pawankrd-cosmosrp-12b-v16-mkmlizer
%s, retrying in %s seconds...
Starting job with name pawankrd-cosmosrp-12b-v16-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v16-mkmlizer to finish
pawankrd-cosmosrp-12b-v16-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ _____ __ __ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ /___/ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Version: 0.11.12 ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v16-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
pawankrd-cosmosrp-12b-v16-mkmlizer: Downloaded to shared memory in 27.971s
pawankrd-cosmosrp-12b-v16-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp5s8hrxb_, device:0
pawankrd-cosmosrp-12b-v16-mkmlizer: Saving flywheel model at /dev/shm/model_cache
pawankrd-cosmosrp-12b-v16-mkmlizer: quantized model in 38.711s
pawankrd-cosmosrp-12b-v16-mkmlizer: Processed model PawanKrd/CosmosRP-12b in 66.683s
pawankrd-cosmosrp-12b-v16-mkmlizer: creating bucket guanaco-mkml-models
pawankrd-cosmosrp-12b-v16-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
pawankrd-cosmosrp-12b-v16-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v16
pawankrd-cosmosrp-12b-v16-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v16/config.json
pawankrd-cosmosrp-12b-v16-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v16/special_tokens_map.json
pawankrd-cosmosrp-12b-v16-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v16/tokenizer_config.json
pawankrd-cosmosrp-12b-v16-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v16/tokenizer.json
pawankrd-cosmosrp-12b-v16-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v16/flywheel_model.0.safetensors
pawankrd-cosmosrp-12b-v16-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:12, 28.04it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:07, 45.16it/s]
Loading 0: 5%|▍ | 17/363 [00:00<00:08, 42.75it/s]
Loading 0: 6%|▌ | 22/363 [00:00<00:07, 42.89it/s]
Loading 0: 7%|▋ | 27/363 [00:00<00:07, 43.22it/s]
Loading 0: 9%|▉ | 32/363 [00:00<00:10, 33.10it/s]
Loading 0: 11%|█ | 39/363 [00:00<00:07, 40.71it/s]
Loading 0: 12%|█▏ | 44/363 [00:01<00:07, 40.48it/s]
Loading 0: 13%|█▎ | 49/363 [00:01<00:07, 39.69it/s]
Loading 0: 15%|█▍ | 54/363 [00:01<00:07, 40.96it/s]
Loading 0: 16%|█▋ | 59/363 [00:01<00:07, 42.96it/s]
Loading 0: 18%|█▊ | 64/363 [00:01<00:11, 25.22it/s]
Loading 0: 20%|█▉ | 71/363 [00:01<00:08, 32.57it/s]
Loading 0: 21%|██ | 76/363 [00:02<00:08, 33.04it/s]
Loading 0: 22%|██▏ | 81/363 [00:02<00:08, 34.46it/s]
Loading 0: 24%|██▎ | 86/363 [00:02<00:07, 36.48it/s]
Loading 0: 25%|██▌ | 91/363 [00:02<00:08, 31.13it/s]
Loading 0: 27%|██▋ | 98/363 [00:02<00:06, 38.12it/s]
Loading 0: 28%|██▊ | 103/363 [00:02<00:06, 38.10it/s]
Loading 0: 30%|██▉ | 108/363 [00:02<00:06, 40.12it/s]
Loading 0: 31%|███ | 113/363 [00:03<00:07, 33.36it/s]
Loading 0: 33%|███▎ | 118/363 [00:03<00:07, 32.36it/s]
Loading 0: 34%|███▍ | 125/363 [00:03<00:06, 38.17it/s]
Loading 0: 36%|███▌ | 130/363 [00:03<00:06, 36.07it/s]
Loading 0: 37%|███▋ | 134/363 [00:03<00:06, 36.35it/s]
Loading 0: 38%|███▊ | 138/363 [00:03<00:06, 35.13it/s]
Loading 0: 39%|███▉ | 142/363 [00:04<00:08, 25.65it/s]
Loading 0: 40%|███▉ | 145/363 [00:04<00:08, 25.13it/s]
Loading 0: 41%|████ | 149/363 [00:04<00:08, 25.08it/s]
Loading 0: 42%|████▏ | 154/363 [00:04<00:06, 30.16it/s]
Loading 0: 44%|████▎ | 158/363 [00:04<00:07, 27.61it/s]
Loading 0: 45%|████▍ | 163/363 [00:04<00:06, 32.46it/s]
Loading 0: 46%|████▌ | 167/363 [00:04<00:06, 30.12it/s]
Loading 0: 47%|████▋ | 172/363 [00:05<00:05, 34.15it/s]
Loading 0: 48%|████▊ | 176/363 [00:05<00:05, 31.48it/s]
Loading 0: 50%|████▉ | 181/363 [00:05<00:05, 35.50it/s]
Loading 0: 51%|█████ | 185/363 [00:05<00:05, 32.29it/s]
Loading 0: 52%|█████▏ | 190/363 [00:05<00:04, 36.27it/s]
Loading 0: 53%|█████▎ | 194/363 [00:05<00:05, 32.57it/s]
Loading 0: 55%|█████▍ | 199/363 [00:05<00:04, 36.36it/s]
Loading 0: 56%|█████▌ | 203/363 [00:05<00:04, 32.45it/s]
Loading 0: 58%|█████▊ | 210/363 [00:06<00:03, 38.96it/s]
Loading 0: 59%|█████▉ | 215/363 [00:06<00:03, 37.58it/s]
Loading 0: 60%|██████ | 219/363 [00:06<00:04, 35.78it/s]
Loading 0: 61%|██████▏ | 223/363 [00:06<00:05, 25.40it/s]
Loading 0: 62%|██████▏ | 226/363 [00:06<00:05, 25.20it/s]
Loading 0: 63%|██████▎ | 230/363 [00:06<00:05, 25.64it/s]
Loading 0: 65%|██████▌ | 237/363 [00:07<00:03, 33.26it/s]
Loading 0: 66%|██████▋ | 241/363 [00:07<00:03, 32.67it/s]
Loading 0: 67%|██████▋ | 245/363 [00:07<00:03, 34.14it/s]
Loading 0: 69%|██████▊ | 249/363 [00:07<00:03, 29.86it/s]
Loading 0: 70%|██████▉ | 253/363 [00:07<00:03, 31.84it/s]
Loading 0: 71%|███████ | 257/363 [00:07<00:03, 30.32it/s]
Loading 0: 73%|███████▎ | 264/363 [00:07<00:02, 36.64it/s]
Loading 0: 74%|███████▍ | 268/363 [00:07<00:02, 34.72it/s]
Loading 0: 75%|███████▌ | 273/363 [00:08<00:02, 36.82it/s]
Loading 0: 76%|███████▋ | 277/363 [00:08<00:02, 35.58it/s]
Loading 0: 78%|███████▊ | 282/363 [00:08<00:02, 38.17it/s]
Loading 0: 79%|███████▉ | 286/363 [00:08<00:02, 36.48it/s]
Loading 0: 80%|████████ | 291/363 [00:08<00:01, 37.58it/s]
Loading 0: 81%|████████▏ | 295/363 [00:08<00:01, 34.94it/s]
Loading 0: 82%|████████▏ | 299/363 [00:08<00:01, 35.05it/s]
Loading 0: 84%|████████▎ | 304/363 [00:15<00:27, 2.11it/s]
Loading 0: 85%|████████▍ | 307/363 [00:15<00:21, 2.64it/s]
Loading 0: 86%|████████▌ | 312/363 [00:15<00:13, 3.87it/s]
Loading 0: 88%|████████▊ | 319/363 [00:16<00:07, 6.27it/s]
Loading 0: 89%|████████▉ | 323/363 [00:16<00:05, 7.91it/s]
Loading 0: 91%|█████████ | 329/363 [00:16<00:03, 11.24it/s]
Loading 0: 92%|█████████▏| 335/363 [00:16<00:01, 14.71it/s]
Loading 0: 94%|█████████▎| 340/363 [00:16<00:01, 17.82it/s]
Loading 0: 96%|█████████▌| 347/363 [00:16<00:00, 23.82it/s]
Loading 0: 97%|█████████▋| 352/363 [00:16<00:00, 27.18it/s]
Loading 0: 98%|█████████▊| 357/363 [00:17<00:00, 25.75it/s]
Job pawankrd-cosmosrp-12b-v16-mkmlizer completed after 94.41s with status: succeeded
Stopping job with name pawankrd-cosmosrp-12b-v16-mkmlizer
Pipeline stage MKMLizer completed in 150.51s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.29s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service pawankrd-cosmosrp-12b-v16
Waiting for inference service pawankrd-cosmosrp-12b-v16 to be ready
Inference service pawankrd-cosmosrp-12b-v16 ready after 131.2971155643463s
Pipeline stage MKMLDeployer completed in 132.26s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.032916784286499s
Received healthy response to inference request in 1.7811846733093262s
Received healthy response to inference request in 2.141120672225952s
Received healthy response to inference request in 1.4758636951446533s
Received healthy response to inference request in 1.4030630588531494s
5 requests
0 failed requests
5th percentile: 1.41762318611145
10th percentile: 1.432183313369751
20th percentile: 1.4613035678863526
30th percentile: 1.5369278907775878
40th percentile: 1.6590562820434571
50th percentile: 1.7811846733093262
60th percentile: 1.8818775177001954
70th percentile: 1.9825703620910644
80th percentile: 2.0545575618743896
90th percentile: 2.097839117050171
95th percentile: 2.1194798946380615
99th percentile: 2.136792516708374
mean time: 1.766829776763916
Pipeline stage StressChecker completed in 10.70s
Shutdown handler de-registered
pawankrd-cosmosrp-12b_v16 status is now deployed due to DeploymentManager action
pawankrd-cosmosrp-12b_v16 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'248a0a41-4787-48eb-bf7e-d7977c418075, 5a9187f5-0b1c-4f64-bd2c-92fb90305696\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:18 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3676611d-c957-4fc0-a497-350a9a81ed52, aeb2962d-48ea-496a-8274-67b8b412ad21\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f67bd716-d6c7-449d-b455-850236dddd34, 5830b016-0b73-4d0e-bd78-e49c431bc8db\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'03422288-db26-4efd-9813-84d131c720cc, d0c93128-af50-4bf9-a1dd-017e2fc108a4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'82f65e33-01e8-4eb5-841b-4b44a6d3d309, 42019e2d-fd28-4f86-81b8-d6b9d0be232f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:36:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1da05394-473f-42b5-a2fb-96972e736f9c, 0fbdd5bf-20e7-490c-8369-23bc836ff877\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'741c7b00-a11e-41be-b746-9902fff0af29, 15bbb6d8-ae72-49a5-a876-37b521b919ea\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'92f7b121-3f3c-4bac-b884-6f8a7bd33fd3, 3b53355c-5413-4a40-8a2c-ea0097f41a96\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c6746a89-8652-476b-a9bb-81c2e37ee81a, aadd1a16-9896-458b-94ba-de952427ea89\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'84f81adf-80e1-4fb9-9260-1b07b675bfbb, 3a5f6547-273b-40d1-acf4-9d28565156b0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c30e4c99-1a4d-475b-bcd7-33b2c732ef88, 2bfe93d3-e9cf-4faa-8ae4-de6863c0f6d7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'254a5c3e-8bb2-42c4-bad4-6373713f542c, 19e37482-de31-4939-90cb-c033b5b7752b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:18 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a701c727-11b0-4d9b-ba81-75b0b24ae254, 7b6f28ea-ef94-457d-9c3e-656a0f035dc8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b141b727-6438-46d7-9b3a-141bc801cdc0, c2e7b6ef-4521-46be-a0a8-5484dc53b333\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'44f21b1c-93eb-4f98-bb4e-3ef5920a42bf, 4436261d-1c30-4e05-87d0-c664ceb0d18c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b62a9863-70ba-4c2c-8d56-785bbd5548ca, 508f3c1c-e43e-41a5-a967-6efb19058e70\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:25 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bba6a546-091d-4cfe-b10b-c3d721ed3d14, 0062389a-64b9-4c7a-9fb2-4c1d9565be51\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:22 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4e421fb8-6e2a-4fe0-b0e9-8e1e765d3eb0, e56ba4e4-1a3e-46a2-9b67-e4f318688db1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c27aa5ae-1f88-4915-b9e1-249008dbcb06, b46155e4-cf20-4d33-b54d-a2e6ef9b34db\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'586c9485-270c-47d0-abff-2800cda641a7, aeecba0a-9f52-446b-bdfa-975c6914e662\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6af28a07-5b4c-4c23-86a4-c8790c1b52dc, e928b285-e7aa-4fbd-b8e9-16e52d5f0914\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'879789ee-f9c7-4977-9727-3204686d5a3f, 81b7da0a-ba03-48c5-87ef-c8cb574a14a9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'77f2f9ef-6356-4780-bce4-b80255bbea26, 9f6f5b44-6837-49dd-a3ee-f3dd8632aa36\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:31 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2ec7dda4-cc5d-4598-807b-dd00dd30db9c, 4bda23ce-879c-4191-b393-ebd44437b04b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b8d2f700-ee48-4071-83a6-3f12a2cafdf8, 7448d5b5-5206-48a3-97c6-c0deecb3e86e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:34 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'41f99ae1-d5dd-4463-8b45-9db4ccdfce36, 2463196f-7ebd-476e-a4f4-d79d4a7bcd9b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:37 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v16
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v16 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ff6a27c2-b9a8-4a96-8022-885912bd9681, 3b9bf6a5-4140-4333-8c8d-70419bd7f48e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
pawankrd-cosmosrp-12b_v16 status is now torndown due to DeploymentManager action