Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rica40325-mistral-test1-v3-mkmlizer
Waiting for job on rica40325-mistral-test1-v3-mkmlizer to finish
rica40325-mistral-test1-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rica40325-mistral-test1-v3-mkmlizer: ║ _____ __ __ ║
rica40325-mistral-test1-v3-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
rica40325-mistral-test1-v3-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
rica40325-mistral-test1-v3-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
rica40325-mistral-test1-v3-mkmlizer: ║ /___/ ║
rica40325-mistral-test1-v3-mkmlizer: ║ ║
rica40325-mistral-test1-v3-mkmlizer: ║ Version: 0.11.12 ║
rica40325-mistral-test1-v3-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
rica40325-mistral-test1-v3-mkmlizer: ║ https://mk1.ai ║
rica40325-mistral-test1-v3-mkmlizer: ║ ║
rica40325-mistral-test1-v3-mkmlizer: ║ The license key for the current software has been verified as ║
rica40325-mistral-test1-v3-mkmlizer: ║ belonging to: ║
rica40325-mistral-test1-v3-mkmlizer: ║ ║
rica40325-mistral-test1-v3-mkmlizer: ║ Chai Research Corp. ║
rica40325-mistral-test1-v3-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rica40325-mistral-test1-v3-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
rica40325-mistral-test1-v3-mkmlizer: ║ ║
rica40325-mistral-test1-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rica40325-mistral-test1-v3-mkmlizer: Downloaded to shared memory in 57.860s
rica40325-mistral-test1-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp41zff991, device:0
rica40325-mistral-test1-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rica40325-mistral-test1-v3-mkmlizer: quantized model in 40.388s
rica40325-mistral-test1-v3-mkmlizer: Processed model rica40325/mistral-test1 in 98.249s
rica40325-mistral-test1-v3-mkmlizer: creating bucket guanaco-mkml-models
rica40325-mistral-test1-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rica40325-mistral-test1-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rica40325-mistral-test1-v3
rica40325-mistral-test1-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rica40325-mistral-test1-v3/config.json
rica40325-mistral-test1-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rica40325-mistral-test1-v3/special_tokens_map.json
rica40325-mistral-test1-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rica40325-mistral-test1-v3/tokenizer_config.json
rica40325-mistral-test1-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rica40325-mistral-test1-v3/tokenizer.json
rica40325-mistral-test1-v3-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rica40325-mistral-test1-v3/flywheel_model.0.safetensors
rica40325-mistral-test1-v3-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:15, 23.39it/s]
Loading 0: 3%|▎ | 10/363 [00:00<00:12, 28.81it/s]
Loading 0: 4%|▍ | 14/363 [00:00<00:13, 25.29it/s]
Loading 0: 6%|▌ | 20/363 [00:00<00:10, 34.28it/s]
Loading 0: 7%|▋ | 24/363 [00:00<00:13, 24.72it/s]
Loading 0: 8%|▊ | 28/363 [00:01<00:13, 24.89it/s]
Loading 0: 9%|▉ | 32/363 [00:01<00:13, 24.08it/s]
Loading 0: 11%|█ | 39/363 [00:01<00:10, 31.24it/s]
Loading 0: 12%|█▏ | 43/363 [00:01<00:10, 30.49it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:09, 32.46it/s]
Loading 0: 14%|█▍ | 52/363 [00:01<00:10, 30.37it/s]
Loading 0: 15%|█▌ | 56/363 [00:01<00:10, 30.58it/s]
Loading 0: 17%|█▋ | 61/363 [00:02<00:11, 26.93it/s]
Loading 0: 18%|█▊ | 64/363 [00:02<00:12, 23.61it/s]
Loading 0: 20%|█▉ | 71/363 [00:02<00:09, 30.26it/s]
Loading 0: 21%|██ | 75/363 [00:02<00:09, 29.37it/s]
Loading 0: 22%|██▏ | 79/363 [00:02<00:09, 28.62it/s]
Loading 0: 23%|██▎ | 84/363 [00:02<00:09, 30.81it/s]
Loading 0: 24%|██▍ | 88/363 [00:03<00:09, 29.52it/s]
Loading 0: 26%|██▌ | 93/363 [00:03<00:08, 31.85it/s]
Loading 0: 27%|██▋ | 97/363 [00:03<00:08, 30.34it/s]
Loading 0: 28%|██▊ | 101/363 [00:03<00:11, 23.80it/s]
Loading 0: 29%|██▊ | 104/363 [00:03<00:12, 21.38it/s]
Loading 0: 31%|███ | 111/363 [00:03<00:08, 28.24it/s]
Loading 0: 32%|███▏ | 115/363 [00:04<00:08, 28.06it/s]
Loading 0: 33%|███▎ | 120/363 [00:04<00:07, 30.86it/s]
Loading 0: 34%|███▍ | 124/363 [00:04<00:08, 29.85it/s]
Loading 0: 36%|███▌ | 129/363 [00:04<00:07, 32.01it/s]
Loading 0: 37%|███▋ | 133/363 [00:04<00:07, 29.79it/s]
Loading 0: 38%|███▊ | 137/363 [00:04<00:07, 29.98it/s]
Loading 0: 39%|███▉ | 142/363 [00:05<00:08, 26.36it/s]
Loading 0: 40%|███▉ | 145/363 [00:05<00:08, 25.01it/s]
Loading 0: 41%|████ | 149/363 [00:05<00:08, 24.01it/s]
Loading 0: 43%|████▎ | 156/363 [00:05<00:06, 30.71it/s]
Loading 0: 44%|████▍ | 160/363 [00:05<00:06, 29.68it/s]
Loading 0: 45%|████▌ | 165/363 [00:05<00:06, 31.89it/s]
Loading 0: 47%|████▋ | 169/363 [00:05<00:06, 30.27it/s]
Loading 0: 48%|████▊ | 174/363 [00:06<00:05, 32.24it/s]
Loading 0: 49%|████▉ | 178/363 [00:06<00:06, 30.63it/s]
Loading 0: 50%|█████ | 182/363 [00:06<00:07, 24.30it/s]
Loading 0: 51%|█████ | 185/363 [00:06<00:08, 21.86it/s]
Loading 0: 53%|█████▎ | 192/363 [00:06<00:05, 28.69it/s]
Loading 0: 54%|█████▍ | 196/363 [00:06<00:05, 28.48it/s]
Loading 0: 55%|█████▌ | 201/363 [00:07<00:05, 31.19it/s]
Loading 0: 56%|█████▋ | 205/363 [00:07<00:05, 30.07it/s]
Loading 0: 58%|█████▊ | 210/363 [00:07<00:04, 32.36it/s]
Loading 0: 59%|█████▉ | 214/363 [00:07<00:04, 30.64it/s]
Loading 0: 60%|██████ | 218/363 [00:07<00:04, 30.42it/s]
Loading 0: 61%|██████▏ | 223/363 [00:07<00:05, 26.39it/s]
Loading 0: 62%|██████▏ | 226/363 [00:08<00:05, 24.95it/s]
Loading 0: 63%|██████▎ | 230/363 [00:08<00:05, 23.82it/s]
Loading 0: 65%|██████▌ | 237/363 [00:08<00:04, 30.56it/s]
Loading 0: 66%|██████▋ | 241/363 [00:08<00:04, 29.80it/s]
Loading 0: 68%|██████▊ | 246/363 [00:08<00:03, 31.84it/s]
Loading 0: 69%|██████▉ | 250/363 [00:08<00:03, 30.79it/s]
Loading 0: 70%|███████ | 255/363 [00:08<00:03, 32.61it/s]
Loading 0: 71%|███████▏ | 259/363 [00:09<00:03, 30.86it/s]
Loading 0: 72%|███████▏ | 263/363 [00:09<00:03, 25.37it/s]
Loading 0: 73%|███████▎ | 266/363 [00:09<00:04, 22.57it/s]
Loading 0: 75%|███████▌ | 273/363 [00:09<00:03, 29.56it/s]
Loading 0: 76%|███████▋ | 277/363 [00:09<00:02, 28.97it/s]
Loading 0: 78%|███████▊ | 282/363 [00:09<00:02, 31.71it/s]
Loading 0: 79%|███████▉ | 286/363 [00:10<00:02, 30.61it/s]
Loading 0: 80%|████████ | 291/363 [00:10<00:02, 32.73it/s]
Loading 0: 81%|████████▏ | 295/363 [00:10<00:02, 30.70it/s]
Loading 0: 82%|████████▏ | 299/363 [00:10<00:02, 30.55it/s]
Loading 0: 84%|████████▎ | 304/363 [00:10<00:02, 25.70it/s]
Loading 0: 85%|████████▍ | 307/363 [00:10<00:02, 24.54it/s]
Loading 0: 86%|████████▌ | 311/363 [00:11<00:02, 23.31it/s]
Loading 0: 88%|████████▊ | 318/363 [00:11<00:01, 30.41it/s]
Loading 0: 89%|████████▊ | 322/363 [00:11<00:01, 29.77it/s]
Loading 0: 90%|█████████ | 327/363 [00:11<00:01, 32.32it/s]
Loading 0: 91%|█████████ | 331/363 [00:11<00:01, 31.14it/s]
Loading 0: 93%|█████████▎| 336/363 [00:11<00:00, 33.06it/s]
Loading 0: 94%|█████████▎| 340/363 [00:11<00:00, 31.17it/s]
Loading 0: 95%|█████████▍| 344/363 [00:18<00:09, 2.01it/s]
Loading 0: 96%|█████████▌| 348/363 [00:18<00:05, 2.70it/s]
Loading 0: 97%|█████████▋| 353/363 [00:19<00:02, 3.91it/s]
Loading 0: 98%|█████████▊| 357/363 [00:19<00:01, 5.09it/s]
Job rica40325-mistral-test1-v3-mkmlizer completed after 124.11s with status: succeeded
Stopping job with name rica40325-mistral-test1-v3-mkmlizer
Pipeline stage MKMLizer completed in 124.84s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.07s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rica40325-mistral-test1-v3
Waiting for inference service rica40325-mistral-test1-v3 to be ready
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name pawankrd-cosmosrp-12b-v7-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v7-mkmlizer to finish
pawankrd-cosmosrp-12b-v7-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ _____ __ __ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ /___/ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Version: 0.11.12 ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
Shutdown handler not registered because Python interpreter is not running in the main thread
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ ║
run pipeline %s
pawankrd-cosmosrp-12b-v7-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name pawankrd-cosmosrp-12b-v8-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v8-mkmlizer to finish
pawankrd-cosmosrp-12b-v8-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ _____ __ __ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ /___/ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Version: 0.11.12 ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name pawankrd-cosmosrp-12b-v9-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v9-mkmlizer to finish
Inference service rica40325-mistral-test1-v3 ready after 140.36271119117737s
Pipeline stage MKMLDeployer completed in 140.77s
run pipeline stage %s
Running pipeline stage StressChecker
pawankrd-cosmosrp-12b-v9-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ _____ __ __ ║
Shutdown handler not registered because Python interpreter is not running in the main thread
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
run pipeline %s
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
run pipeline stage %s
Received healthy response to inference request in 2.6815924644470215s
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
Running pipeline stage MKMLizer
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ /___/ ║
Starting job with name pawankrd-cosmosrp-12b-v10-mkmlizer
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Version: 0.11.12 ║
Waiting for job on pawankrd-cosmosrp-12b-v10-mkmlizer to finish
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ ║
Received healthy response to inference request in 1.7252769470214844s
pawankrd-cosmosrp-12b-v9-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Received healthy response to inference request in 1.7261509895324707s
Received healthy response to inference request in 2.1664175987243652s
pawankrd-cosmosrp-12b-v7-mkmlizer: Downloaded to shared memory in 30.174s
pawankrd-cosmosrp-12b-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp0u3w3ygc, device:0
pawankrd-cosmosrp-12b-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Received healthy response to inference request in 2.0086138248443604s
5 requests
0 failed requests
5th percentile: 1.7254517555236817
10th percentile: 1.725626564025879
20th percentile: 1.7259761810302734
30th percentile: 1.7826435565948486
40th percentile: 1.8956286907196045
50th percentile: 2.0086138248443604
60th percentile: 2.0717353343963625
70th percentile: 2.134856843948364
80th percentile: 2.2694525718688965
90th percentile: 2.475522518157959
95th percentile: 2.5785574913024902
99th percentile: 2.6609854698181152
mean time: 2.0616103649139403
Pipeline stage StressChecker completed in 13.11s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
pawankrd-cosmosrp-12b-v10-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ _____ __ __ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ /___/ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Version: 0.11.12 ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.73s
Shutdown handler de-registered
rica40325-mistral-test1_v3 status is now deployed due to DeploymentManager action
rica40325-mistral-test1_v3 status is now inactive due to admin request
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4aeb8bd1-3bd4-4bba-81a6-a4918ebe5e04, a1abf08a-136a-43ce-a332-38a282c7bc12\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'768b3ff0-4a16-4563-999e-73be997f0430, 8c5174fa-4969-4035-ab52-ce5038f471f0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'33839248-30e5-4578-a7f0-e74629a97d60, 0a77d864-7e03-482d-9b11-fa0bd83ab7d8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6413c6dc-4f43-4550-9102-b1325843dd6f, 98dd9707-630f-45b6-a2a1-4faee6120beb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8c39e79a-6c71-4e99-9c75-049374609885, 2d8176d1-f83e-4e5e-8d3f-1e7daeee9013\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:37:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'49582e7b-ff37-4aec-9524-93085fcc9dd8, f96726eb-08b1-4b4d-82ce-9a565c2ecad9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bb9f719c-667e-449c-ae84-762c8539a85a, f4d8bb8c-0460-43eb-8ffa-12be579ebf17\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bb1f4311-2ee2-4c7e-b9a2-868ea6d520b2, e42a1001-be24-42ce-9c53-819d7655c87e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'115dbc61-172d-42ae-8580-8eec54cb76fc, 557f8961-2638-4228-a85a-84b1174ff957\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8dd794b0-82ff-4a28-8108-9a40e4a8aa32, b5afe37d-b7e5-4c0f-9e2b-031e5fe79f82\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bdd5e640-2509-49b8-88d2-45f3b987d60e, d8230fc3-53e5-4363-9753-7a4579254183\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:23 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'19220cba-4f5f-41b5-8fcf-51d88240a16f, 787f2e2f-061d-4108-81cd-c468bbeb1103\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:22 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9d27dc28-d279-4d9d-a411-7ffe03f9bd86, 260c40e1-961c-428e-9ae5-95bb86cb8265\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b7f3cff9-f366-41ef-96c1-34fcbe8c3bfa, c47eb707-97f8-4ec6-8e0b-c5519bd983b4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:23 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'434f87ba-e1e6-4820-a8e7-1357e89430c8, 4a26b36f-3545-41fa-b648-74e87f5ccbe4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:23 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f7da39dc-9b8c-42be-a9f7-f391d6e19107, 01ee6d8f-310b-4201-bfa9-6fe75d1b8ad1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8df90c2d-2d0c-4dbe-9c49-20c4a6f651f2, 2347e0a2-c317-4934-a6c5-8564be8049fd\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2ddeffc5-6033-4dbd-8679-cd9d1f3a34c2, 473b8d15-4805-49da-9fd1-f985562ec0f5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'05a05772-85ff-4c26-8535-5e519d1a40ee, 69b319d7-bec7-4e98-ad1f-395863ffd038\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'40a51496-35c9-45bc-9609-30fea6492c8e, 61be3a8c-dfe6-45ef-b660-9291c815c87a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a7d4002b-2baa-4998-9a9a-2dad299fb08d, 11a66436-f944-4419-86a7-c86751155409\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'adc7d384-93df-4da5-acad-b430399e415f, a777c0be-50eb-4f91-9616-ffb708565eda\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fd92837b-e4bb-47cb-af9e-1f230673bdc8, 42d4a0ff-aeec-4c71-9727-b7815b365410\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d506fe5e-5dcb-4c99-8abd-051345528725, 7fc90a45-c40b-4b3c-b63a-2595fe5e5b76\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'eaec9aac-e744-49c6-8159-98b0aeeb98d6, 5b045c6a-e4aa-48c2-be49-ea5db9fed01d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:37 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'936c447f-9e76-41ff-8628-79c2fa0a091a, da766650-dbcb-4e8e-94e6-87a58e660c3c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:41 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a18ca9bd-3bd0-404b-b137-7613c8dd9f06, 4866f491-0c0e-45b4-a309-443d961e166e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
rica40325-mistral-test1_v3 status is now torndown due to DeploymentManager action