Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name pawankrd-cosmosrp-12b-v7-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v7-mkmlizer to finish
pawankrd-cosmosrp-12b-v7-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ _____ __ __ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ /___/ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Version: 0.11.12 ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
Shutdown handler not registered because Python interpreter is not running in the main thread
pawankrd-cosmosrp-12b-v7-mkmlizer: ║ ║
run pipeline %s
pawankrd-cosmosrp-12b-v7-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name pawankrd-cosmosrp-12b-v8-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v8-mkmlizer to finish
pawankrd-cosmosrp-12b-v8-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ _____ __ __ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ /___/ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Version: 0.11.12 ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v8-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name pawankrd-cosmosrp-12b-v9-mkmlizer
Waiting for job on pawankrd-cosmosrp-12b-v9-mkmlizer to finish
Inference service rica40325-mistral-test1-v3 ready after 140.36271119117737s
Pipeline stage MKMLDeployer completed in 140.77s
run pipeline stage %s
Running pipeline stage StressChecker
pawankrd-cosmosrp-12b-v9-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ _____ __ __ ║
Shutdown handler not registered because Python interpreter is not running in the main thread
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
run pipeline %s
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
run pipeline stage %s
Received healthy response to inference request in 2.6815924644470215s
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
Running pipeline stage MKMLizer
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ /___/ ║
Starting job with name pawankrd-cosmosrp-12b-v10-mkmlizer
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Version: 0.11.12 ║
Waiting for job on pawankrd-cosmosrp-12b-v10-mkmlizer to finish
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pawankrd-cosmosrp-12b-v9-mkmlizer: ║ ║
Received healthy response to inference request in 1.7252769470214844s
pawankrd-cosmosrp-12b-v9-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Received healthy response to inference request in 1.7261509895324707s
Received healthy response to inference request in 2.1664175987243652s
pawankrd-cosmosrp-12b-v7-mkmlizer: Downloaded to shared memory in 30.174s
pawankrd-cosmosrp-12b-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp0u3w3ygc, device:0
pawankrd-cosmosrp-12b-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Received healthy response to inference request in 2.0086138248443604s
5 requests
0 failed requests
5th percentile: 1.7254517555236817
10th percentile: 1.725626564025879
20th percentile: 1.7259761810302734
30th percentile: 1.7826435565948486
40th percentile: 1.8956286907196045
50th percentile: 2.0086138248443604
60th percentile: 2.0717353343963625
70th percentile: 2.134856843948364
80th percentile: 2.2694525718688965
90th percentile: 2.475522518157959
95th percentile: 2.5785574913024902
99th percentile: 2.6609854698181152
mean time: 2.0616103649139403
Pipeline stage StressChecker completed in 13.11s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
pawankrd-cosmosrp-12b-v10-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ _____ __ __ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ /___/ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Version: 0.11.12 ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ https://mk1.ai ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ The license key for the current software has been verified as ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ belonging to: ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Chai Research Corp. ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ║ ║
pawankrd-cosmosrp-12b-v10-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.73s
Shutdown handler de-registered
rica40325-mistral-test1_v3 status is now deployed due to DeploymentManager action
pawankrd-cosmosrp-12b-v8-mkmlizer: Downloaded to shared memory in 28.356s
pawankrd-cosmosrp-12b-v8-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpsiadektx, device:0
pawankrd-cosmosrp-12b-v8-mkmlizer: Saving flywheel model at /dev/shm/model_cache
pawankrd-cosmosrp-12b-v9-mkmlizer: Downloaded to shared memory in 28.110s
pawankrd-cosmosrp-12b-v9-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp14wwp37d, device:0
pawankrd-cosmosrp-12b-v9-mkmlizer: Saving flywheel model at /dev/shm/model_cache
pawankrd-cosmosrp-12b-v10-mkmlizer: Downloaded to shared memory in 27.617s
pawankrd-cosmosrp-12b-v10-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp02bp274e, device:0
pawankrd-cosmosrp-12b-v10-mkmlizer: Saving flywheel model at /dev/shm/model_cache
pawankrd-cosmosrp-12b-v7-mkmlizer: quantized model in 36.225s
pawankrd-cosmosrp-12b-v7-mkmlizer: Processed model PawanKrd/CosmosRP-12b in 66.399s
pawankrd-cosmosrp-12b-v7-mkmlizer: creating bucket guanaco-mkml-models
pawankrd-cosmosrp-12b-v7-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
pawankrd-cosmosrp-12b-v7-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v7
pawankrd-cosmosrp-12b-v7-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v7/config.json
pawankrd-cosmosrp-12b-v7-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v7/special_tokens_map.json
pawankrd-cosmosrp-12b-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v7/tokenizer_config.json
pawankrd-cosmosrp-12b-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v7/tokenizer.json
pawankrd-cosmosrp-12b-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v7/flywheel_model.0.safetensors
pawankrd-cosmosrp-12b-v7-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:12, 29.17it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:07, 47.30it/s]
Loading 0: 5%|▍ | 18/363 [00:00<00:07, 46.33it/s]
Loading 0: 6%|▋ | 23/363 [00:00<00:09, 35.31it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:07, 45.62it/s]
Loading 0: 10%|█ | 37/363 [00:00<00:07, 43.99it/s]
Loading 0: 12%|█▏ | 42/363 [00:00<00:07, 43.47it/s]
Loading 0: 13%|█▎ | 49/363 [00:01<00:06, 49.25it/s]
Loading 0: 15%|█▌ | 55/363 [00:01<00:06, 47.11it/s]
Loading 0: 17%|█▋ | 61/363 [00:01<00:08, 35.54it/s]
Loading 0: 18%|█▊ | 66/363 [00:01<00:08, 36.17it/s]
Loading 0: 20%|█▉ | 72/363 [00:01<00:07, 40.21it/s]
Loading 0: 21%|██ | 77/363 [00:01<00:06, 41.91it/s]
Loading 0: 23%|██▎ | 82/363 [00:02<00:07, 36.40it/s]
Loading 0: 25%|██▍ | 89/363 [00:02<00:06, 43.41it/s]
Loading 0: 26%|██▌ | 94/363 [00:02<00:06, 42.82it/s]
Loading 0: 27%|██▋ | 99/363 [00:02<00:06, 40.11it/s]
Loading 0: 29%|██▊ | 104/363 [00:02<00:06, 41.79it/s]
Loading 0: 30%|███ | 110/363 [00:02<00:06, 40.50it/s]
Loading 0: 32%|███▏ | 115/363 [00:02<00:05, 41.47it/s]
Loading 0: 33%|███▎ | 120/363 [00:02<00:06, 39.85it/s]
Loading 0: 35%|███▍ | 126/363 [00:03<00:05, 42.81it/s]
Loading 0: 36%|███▌ | 131/363 [00:03<00:05, 43.05it/s]
Loading 0: 37%|███▋ | 136/363 [00:03<00:06, 35.63it/s]
Loading 0: 39%|███▉ | 142/363 [00:03<00:07, 31.27it/s]
Loading 0: 40%|████ | 146/363 [00:03<00:06, 31.73it/s]
Loading 0: 41%|████▏ | 150/363 [00:03<00:06, 30.86it/s]
Loading 0: 43%|████▎ | 156/363 [00:03<00:05, 35.72it/s]
Loading 0: 44%|████▍ | 160/363 [00:04<00:05, 35.46it/s]
Loading 0: 45%|████▌ | 165/363 [00:04<00:05, 38.92it/s]
Loading 0: 47%|████▋ | 170/363 [00:04<00:04, 39.84it/s]
Loading 0: 48%|████▊ | 175/363 [00:04<00:04, 41.19it/s]
Loading 0: 50%|████▉ | 180/363 [00:04<00:04, 42.49it/s]
Loading 0: 51%|█████ | 185/363 [00:04<00:05, 34.54it/s]
Loading 0: 53%|█████▎ | 192/363 [00:04<00:04, 42.10it/s]
Loading 0: 54%|█████▍ | 197/363 [00:04<00:03, 41.67it/s]
Loading 0: 56%|█████▌ | 202/363 [00:05<00:03, 41.95it/s]
Loading 0: 57%|█████▋ | 208/363 [00:05<00:03, 40.97it/s]
Loading 0: 59%|█████▊ | 213/363 [00:05<00:03, 40.08it/s]
Loading 0: 60%|██████ | 218/363 [00:05<00:03, 41.84it/s]
Loading 0: 61%|██████▏ | 223/363 [00:05<00:04, 32.39it/s]
Loading 0: 63%|██████▎ | 227/363 [00:05<00:04, 33.35it/s]
Loading 0: 64%|██████▎ | 231/363 [00:05<00:04, 32.17it/s]
Loading 0: 65%|██████▌ | 237/363 [00:06<00:03, 37.60it/s]
Loading 0: 67%|██████▋ | 242/363 [00:06<00:03, 39.52it/s]
Loading 0: 68%|██████▊ | 247/363 [00:06<00:02, 41.37it/s]
Loading 0: 69%|██████▉ | 252/363 [00:06<00:02, 42.77it/s]
Loading 0: 71%|███████ | 257/363 [00:06<00:03, 34.14it/s]
Loading 0: 73%|███████▎ | 264/363 [00:06<00:02, 40.22it/s]
Loading 0: 74%|███████▍ | 269/363 [00:06<00:02, 40.70it/s]
Loading 0: 75%|███████▌ | 274/363 [00:06<00:02, 41.26it/s]
Loading 0: 77%|███████▋ | 280/363 [00:07<00:02, 41.18it/s]
Loading 0: 79%|███████▊ | 285/363 [00:07<00:01, 40.70it/s]
Loading 0: 80%|████████ | 291/363 [00:07<00:01, 45.09it/s]
Loading 0: 82%|████████▏ | 296/363 [00:07<00:01, 44.53it/s]
Loading 0: 83%|████████▎ | 302/363 [00:07<00:01, 48.18it/s]
Loading 0: 85%|████████▍ | 307/363 [00:14<00:22, 2.54it/s]
Loading 0: 86%|████████▌ | 312/363 [00:14<00:14, 3.46it/s]
Loading 0: 88%|████████▊ | 320/363 [00:14<00:07, 5.54it/s]
Loading 0: 90%|████████▉ | 326/363 [00:14<00:04, 7.46it/s]
Loading 0: 91%|█████████ | 331/363 [00:14<00:03, 9.47it/s]
Loading 0: 93%|█████████▎| 338/363 [00:14<00:01, 13.37it/s]
Loading 0: 95%|█████████▍| 344/363 [00:15<00:01, 16.73it/s]
Loading 0: 96%|█████████▌| 349/363 [00:15<00:00, 19.70it/s]
Loading 0: 98%|█████████▊| 356/363 [00:15<00:00, 25.86it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 29.20it/s]
Job pawankrd-cosmosrp-12b-v7-mkmlizer completed after 94.43s with status: succeeded
Stopping job with name pawankrd-cosmosrp-12b-v7-mkmlizer
Pipeline stage MKMLizer completed in 94.94s
pawankrd-cosmosrp-12b-v8-mkmlizer: quantized model in 36.501s
run pipeline stage %s
pawankrd-cosmosrp-12b-v8-mkmlizer: Processed model PawanKrd/CosmosRP-12b in 64.857s
Running pipeline stage MKMLTemplater
pawankrd-cosmosrp-12b-v8-mkmlizer: creating bucket guanaco-mkml-models
Pipeline stage MKMLTemplater completed in 0.21s
pawankrd-cosmosrp-12b-v8-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
run pipeline stage %s
pawankrd-cosmosrp-12b-v8-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v8
Running pipeline stage MKMLDeployer
pawankrd-cosmosrp-12b-v8-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v8/config.json
Creating inference service pawankrd-cosmosrp-12b-v7
pawankrd-cosmosrp-12b-v8-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v8/special_tokens_map.json
pawankrd-cosmosrp-12b-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v8/tokenizer_config.json
Waiting for inference service pawankrd-cosmosrp-12b-v7 to be ready
pawankrd-cosmosrp-12b-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v8/tokenizer.json
Job pawankrd-cosmosrp-12b-v8-mkmlizer completed after 94.89s with status: succeeded
Stopping job with name pawankrd-cosmosrp-12b-v8-mkmlizer
Pipeline stage MKMLizer completed in 95.50s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service pawankrd-cosmosrp-12b-v8
Waiting for inference service pawankrd-cosmosrp-12b-v8 to be ready
pawankrd-cosmosrp-12b-v9-mkmlizer: quantized model in 37.196s
pawankrd-cosmosrp-12b-v9-mkmlizer: Processed model PawanKrd/CosmosRP-12b in 65.306s
pawankrd-cosmosrp-12b-v9-mkmlizer: creating bucket guanaco-mkml-models
pawankrd-cosmosrp-12b-v9-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
pawankrd-cosmosrp-12b-v9-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v9
pawankrd-cosmosrp-12b-v9-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v9/config.json
pawankrd-cosmosrp-12b-v9-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v9/special_tokens_map.json
pawankrd-cosmosrp-12b-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v9/tokenizer_config.json
pawankrd-cosmosrp-12b-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v9/tokenizer.json
pawankrd-cosmosrp-12b-v9-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:14, 24.56it/s]
Loading 0: 3%|▎ | 10/363 [00:00<00:11, 31.60it/s]
Loading 0: 4%|▍ | 14/363 [00:00<00:11, 30.51it/s]
Loading 0: 6%|▌ | 21/363 [00:00<00:08, 41.99it/s]
Loading 0: 7%|▋ | 26/363 [00:00<00:08, 41.58it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:07, 41.80it/s]
Loading 0: 10%|▉ | 36/363 [00:00<00:08, 39.14it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:10, 30.94it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:08, 38.74it/s]
Loading 0: 15%|█▍ | 53/363 [00:01<00:08, 38.70it/s]
Loading 0: 16%|█▌ | 58/363 [00:01<00:07, 39.74it/s]
Loading 0: 17%|█▋ | 63/363 [00:01<00:11, 25.64it/s]
Loading 0: 18%|█▊ | 67/363 [00:02<00:10, 27.76it/s]
Loading 0: 20%|█▉ | 71/363 [00:02<00:09, 29.63it/s]
Loading 0: 21%|██ | 75/363 [00:02<00:09, 29.28it/s]
Loading 0: 22%|██▏ | 79/363 [00:02<00:09, 31.06it/s]
Loading 0: 23%|██▎ | 83/363 [00:02<00:09, 28.90it/s]
Loading 0: 24%|██▍ | 87/363 [00:02<00:08, 31.39it/s]
Loading 0: 25%|██▌ | 91/363 [00:02<00:09, 29.19it/s]
Loading 0: 26%|██▋ | 96/363 [00:02<00:07, 33.99it/s]
Loading 0: 28%|██▊ | 100/363 [00:03<00:08, 30.69it/s]
Loading 0: 29%|██▉ | 106/363 [00:03<00:07, 35.55it/s]
Loading 0: 31%|███ | 112/363 [00:03<00:06, 39.74it/s]
Loading 0: 32%|███▏ | 117/363 [00:03<00:06, 38.59it/s]
Loading 0: 34%|███▎ | 122/363 [00:03<00:05, 41.30it/s]
Loading 0: 35%|███▍ | 127/363 [00:03<00:06, 35.79it/s]
Loading 0: 37%|███▋ | 135/363 [00:03<00:05, 43.99it/s]
Loading 0: 39%|███▉ | 141/363 [00:03<00:05, 44.33it/s]
Loading 0: 40%|████ | 146/363 [00:04<00:07, 30.96it/s]
Loading 0: 41%|████▏ | 150/363 [00:04<00:06, 31.00it/s]
Loading 0: 43%|████▎ | 157/363 [00:04<00:05, 37.63it/s]
Loading 0: 45%|████▍ | 163/363 [00:04<00:05, 38.65it/s]
Loading 0: 46%|████▋ | 168/363 [00:04<00:05, 38.33it/s]
Loading 0: 48%|████▊ | 174/363 [00:04<00:04, 42.15it/s]
Loading 0: 49%|████▉ | 179/363 [00:05<00:04, 41.74it/s]
Loading 0: 51%|█████ | 184/363 [00:05<00:04, 40.67it/s]
Loading 0: 52%|█████▏ | 190/363 [00:05<00:04, 39.51it/s]
Loading 0: 54%|█████▎ | 195/363 [00:05<00:04, 39.02it/s]
Loading 0: 56%|█████▌ | 202/363 [00:05<00:03, 44.42it/s]
Loading 0: 57%|█████▋ | 207/363 [00:05<00:03, 45.71it/s]
Loading 0: 58%|█████▊ | 212/363 [00:05<00:04, 37.40it/s]
Loading 0: 60%|██████ | 218/363 [00:06<00:03, 39.61it/s]
Loading 0: 61%|██████▏ | 223/363 [00:06<00:04, 29.91it/s]
Loading 0: 63%|██████▎ | 227/363 [00:06<00:04, 29.10it/s]
Loading 0: 64%|██████▎ | 231/363 [00:06<00:04, 28.90it/s]
Loading 0: 66%|██████▌ | 238/363 [00:06<00:03, 35.99it/s]
Loading 0: 67%|██████▋ | 244/363 [00:06<00:03, 38.32it/s]
Loading 0: 69%|██████▊ | 249/363 [00:06<00:02, 38.46it/s]
Loading 0: 70%|███████ | 255/363 [00:07<00:02, 42.66it/s]
Loading 0: 72%|███████▏ | 260/363 [00:07<00:02, 44.31it/s]
Loading 0: 73%|███████▎ | 265/363 [00:07<00:02, 45.68it/s]
Loading 0: 75%|███████▍ | 271/363 [00:07<00:02, 45.14it/s]
Loading 0: 76%|███████▌ | 276/363 [00:07<00:01, 43.55it/s]
Loading 0: 78%|███████▊ | 283/363 [00:07<00:01, 48.15it/s]
Loading 0: 80%|███████▉ | 289/363 [00:07<00:01, 44.65it/s]
Loading 0: 81%|████████ | 294/363 [00:07<00:01, 42.18it/s]
Loading 0: 82%|████████▏ | 299/363 [00:08<00:01, 42.57it/s]
Loading 0: 84%|████████▎ | 304/363 [00:14<00:23, 2.51it/s]
Loading 0: 85%|████████▍ | 308/363 [00:14<00:17, 3.23it/s]
Loading 0: 86%|████████▌ | 312/363 [00:15<00:12, 4.19it/s]
Loading 0: 88%|████████▊ | 319/363 [00:15<00:06, 6.63it/s]
Loading 0: 89%|████████▉ | 324/363 [00:15<00:04, 8.74it/s]
Loading 0: 91%|█████████ | 329/363 [00:15<00:02, 11.38it/s]
Loading 0: 92%|█████████▏| 335/363 [00:15<00:01, 14.87it/s]
Loading 0: 94%|█████████▎| 340/363 [00:15<00:01, 18.03it/s]
Loading 0: 95%|█████████▌| 346/363 [00:15<00:00, 23.29it/s]
Loading 0: 97%|█████████▋| 351/363 [00:15<00:00, 26.60it/s]
Loading 0: 98%|█████████▊| 356/363 [00:16<00:00, 30.17it/s]
Loading 0: 99%|█████████▉| 361/363 [00:16<00:00, 32.83it/s]
pawankrd-cosmosrp-12b-v10-mkmlizer: quantized model in 36.913s
Job pawankrd-cosmosrp-12b-v9-mkmlizer completed after 96.12s with status: succeeded
pawankrd-cosmosrp-12b-v10-mkmlizer: Processed model PawanKrd/CosmosRP-12b in 64.531s
Stopping job with name pawankrd-cosmosrp-12b-v9-mkmlizer
pawankrd-cosmosrp-12b-v10-mkmlizer: creating bucket guanaco-mkml-models
Pipeline stage MKMLizer completed in 96.91s
pawankrd-cosmosrp-12b-v10-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
run pipeline stage %s
pawankrd-cosmosrp-12b-v10-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v10
Running pipeline stage MKMLTemplater
pawankrd-cosmosrp-12b-v10-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v10/config.json
Pipeline stage MKMLTemplater completed in 0.25s
pawankrd-cosmosrp-12b-v10-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v10/special_tokens_map.json
run pipeline stage %s
pawankrd-cosmosrp-12b-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v10/tokenizer_config.json
Running pipeline stage MKMLDeployer
pawankrd-cosmosrp-12b-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/pawankrd-cosmosrp-12b-v10/tokenizer.json
Creating inference service pawankrd-cosmosrp-12b-v9
Waiting for inference service pawankrd-cosmosrp-12b-v9 to be ready
Job pawankrd-cosmosrp-12b-v10-mkmlizer completed after 96.26s with status: succeeded
Stopping job with name pawankrd-cosmosrp-12b-v10-mkmlizer
Pipeline stage MKMLizer completed in 97.18s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service pawankrd-cosmosrp-12b-v10
Waiting for inference service pawankrd-cosmosrp-12b-v10 to be ready
Inference service pawankrd-cosmosrp-12b-v7 ready after 270.58050775527954s
Pipeline stage MKMLDeployer completed in 271.39s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service pawankrd-cosmosrp-12b-v8 ready after 260.6100883483887s
Pipeline stage MKMLDeployer completed in 261.29s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.719313144683838s
Received healthy response to inference request in 2.0898609161376953s
Received healthy response to inference request in 1.5793137550354004s
Inference service pawankrd-cosmosrp-12b-v9 ready after 250.61476516723633s
Pipeline stage MKMLDeployer completed in 251.40s
Received healthy response to inference request in 1.6355342864990234s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.4664082527160645s
Inference service pawankrd-cosmosrp-12b-v10 ready after 240.53126430511475s
Pipeline stage MKMLDeployer completed in 241.29s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.8456628322601318s
Received healthy response to inference request in 1.9360766410827637s
Received healthy response to inference request in 1.9801831245422363s
Received healthy response to inference request in 1.591768741607666s
Received healthy response to inference request in 2.3266780376434326s
Received healthy response to inference request in 1.4782779216766357s
5 requests
Received healthy response to inference request in 1.7102630138397217s
0 failed requests
5th percentile: 1.4687821865081787
10th percentile: 1.471156120300293
20th percentile: 1.4759039878845215
30th percentile: 1.4984850883483887
40th percentile: 1.5388994216918945
50th percentile: 1.5793137550354004
Received healthy response to inference request in 1.707120418548584s
60th percentile: 1.7220189094543457
Received healthy response to inference request in 1.6254703998565674s
5 requests
70th percentile: 1.864724063873291
0 failed requests
80th percentile: 2.0927239418029786
5th percentile: 1.6005218505859375
Received healthy response to inference request in 1.9885213375091553s
90th percentile: 2.406018543243408
10th percentile: 1.609274959564209
95th percentile: 2.562665843963623
20th percentile: 1.626781177520752
99th percentile: 2.687983684539795
30th percentile: 1.6498515129089355
mean time: 1.8358779430389405
40th percentile: 1.6784859657287599
Pipeline stage StressChecker completed in 13.41s
50th percentile: 1.707120418548584
run pipeline stage %s
60th percentile: 1.762537384033203
Received healthy response to inference request in 1.7063863277435303s
Running pipeline stage TriggerMKMLProfilingPipeline
70th percentile: 1.8179543495178223
run_pipeline:run_in_cloud %s
80th percentile: 1.8945024490356446
starting trigger_guanaco_pipeline args=%s
90th percentile: 1.9921816825866698
Received healthy response to inference request in 1.8904786109924316s
95th percentile: 2.041021299362183
99th percentile: 2.0800929927825926
mean time: 1.7739894390106201
Pipeline stage StressChecker completed in 13.85s
run pipeline stage %s
Received healthy response to inference request in 1.9591355323791504s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.89s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 1.8896374702453613s
Shutdown handler de-registered
5 requests
5 requests
pawankrd-cosmosrp-12b_v7 status is now deployed due to DeploymentManager action
pawankrd-cosmosrp-12b_v7 status is now inactive due to admin request
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1409a478-d0b7-4430-b919-b7b72cac5726, 64edf679-fec2-49e4-87bd-17d8a800a849\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'85b3ffe1-8b90-4c36-b700-9458fa4fb74d, 008827d0-d21d-4155-891f-eb0e852df274\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'758f5ce4-2a7a-4a29-b621-840e701abf56, 4c2ff77d-d4ff-4be9-b326-5bcf706efd03\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'996f2b87-12e9-4a1c-ac51-7a8ab5ee4e0b, 6c33e189-c7c2-44b9-bc5e-0769ff879db9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'519e371a-0361-41cc-8155-bae722c3203d, 90e29e20-37e2-46ee-a28d-64f99ba47f33\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:36:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1c6bf6f7-b1dc-45bd-a1eb-b8d1e61ad4ee, 030a5340-4eb8-4cdb-89ce-6583a36aa9c5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c8d88264-8fbe-440e-b44f-9527896c0f3b, e2adcfab-7674-49e2-be52-19c1ebc49073\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ec2e3054-6c93-4cc1-9d72-90dfd82c091a, 67bd3b6e-3603-4e35-83a5-f8f2c5ce2e84\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e831e9fb-6136-4e34-bbe7-c510553386f1, 62c4fa79-8e25-483c-9b68-4fefdfb7bfb0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9b7ba7af-3cc8-4a2e-92ca-7396cdfce858, 376f9fe3-2b34-4f33-8df5-541e9bccaffb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'07271caf-835d-4bbd-b052-5247a4a89e1e, 9e689e32-b3dc-4035-a463-cd7bb0cb1360\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ffe60882-2e63-46de-a609-c988fc4f6f07, ef197e90-604b-4b54-b648-d858c9e98643\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bcaceeca-ad96-446e-bb4e-7a22025df550, 5a546e81-dd5f-4604-8682-e8cc9b382e99\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:22 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e8442860-53ce-4bbe-943d-0a41bf816bb6, 70076a82-f347-4bae-a9c3-0aa85cf7014a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:22 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9bb0fb5f-6215-4719-ac6e-5027f6b02bad, cd82c20f-7a57-41ed-97cd-8d82e1f7b48a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bdb6b610-848f-4980-a7c2-9b5605064058, 6b3c29ed-7381-4d51-bff1-cb3e6bc580af\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3fcfeef7-9a96-4758-803a-f7e4ca55833e, 5221f13b-4e38-4cc5-b139-9fff54484528\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d5900522-893c-4a9b-b1c8-47fe211b0487, 3c2d4e5c-c958-4728-806d-cf7c6c6a26d9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:31 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9ad27d18-cdf9-469d-a92d-1ebb4032fe62, dc120a30-bbc7-4fa0-bbc3-3ee2d8c80af4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9e3af5bb-150c-4886-9024-bc70e573795d, fef97c82-d57f-4f5d-81f0-11e55d8d5bd6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:31 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cc989a30-53d9-43fb-a7d7-8901ce6995c4, b7f9319b-6b54-4186-a567-9ce013825d7b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'df4e47ed-1e20-430c-b143-e7c5e9d382ec, 1b0026b7-e25a-4da6-b0fa-997b81f5c816\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:47 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6cbe2e9b-28c9-46be-aed6-cf44b2994292, a138441a-72ff-4a8a-bca8-5191b479178c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e7c3caa5-26e5-4f7f-9dee-5de240b7aa35, b4a16df8-6bd8-4848-8420-6c142f776b27\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:38 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'819a9a37-495b-48a4-b2f7-0eea392e483e, 87396f5e-3306-43d2-96f1-7a36467d1efa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:36 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7861954f-9add-4ad2-9680-b9fffab3e4d3, 4d2d5218-c4df-4666-a554-685916817b98\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:38 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of pawankrd-cosmosrp-12b_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
%s, retrying in %s seconds...
Checking if service pawankrd-cosmosrp-12b-v7 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f248f054-9be3-4787-8014-3c3ac0f8cac5, ebdabd42-b27e-4c8c-a2b1-7f3307dbeaba\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:37 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
pawankrd-cosmosrp-12b_v7 status is now torndown due to DeploymentManager action