Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer
Waiting for job on chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer to finish
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ _____ __ __ ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ /___/ ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ Version: 0.11.12 ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ https://mk1.ai ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ belonging to: ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ Chai Research Corp. ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ║ ║
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: Downloaded to shared memory in 48.429s
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmphriy02yr, device:0
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: quantized model in 36.228s
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: Processed model ChaiML/Sao10k-MN-12B-Lyra-v4 in 84.657s
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-sao10k-mn-12b-lyra-v4-v1
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-sao10k-mn-12b-lyra-v4-v1/config.json
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-sao10k-mn-12b-lyra-v4-v1/special_tokens_map.json
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-sao10k-mn-12b-lyra-v4-v1/tokenizer_config.json
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-sao10k-mn-12b-lyra-v4-v1/tokenizer.json
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-sao10k-mn-12b-lyra-v4-v1/flywheel_model.0.safetensors
chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 2/363 [00:06<18:30, 3.07s/it]
Loading 0: 2%|▏ | 6/363 [00:06<04:55, 1.21it/s]
Loading 0: 4%|▎ | 13/363 [00:06<01:45, 3.32it/s]
Loading 0: 5%|▍ | 17/363 [00:06<01:11, 4.86it/s]
Loading 0: 6%|▋ | 23/363 [00:06<00:42, 7.91it/s]
Loading 0: 8%|▊ | 28/363 [00:06<00:30, 11.01it/s]
Loading 0: 9%|▉ | 33/363 [00:06<00:24, 13.65it/s]
Loading 0: 11%|█▏ | 41/363 [00:07<00:15, 20.72it/s]
Loading 0: 13%|█▎ | 47/363 [00:07<00:12, 24.62it/s]
Loading 0: 14%|█▍ | 52/363 [00:07<00:11, 27.51it/s]
Loading 0: 16%|█▋ | 59/363 [00:07<00:08, 33.83it/s]
Loading 0: 18%|█▊ | 65/363 [00:07<00:08, 35.10it/s]
Loading 0: 19%|█▉ | 70/363 [00:07<00:08, 35.42it/s]
Loading 0: 21%|██ | 75/363 [00:08<00:11, 24.70it/s]
Loading 0: 22%|██▏ | 79/363 [00:08<00:11, 25.73it/s]
Loading 0: 24%|██▎ | 86/363 [00:08<00:08, 32.47it/s]
Loading 0: 25%|██▌ | 91/363 [00:08<00:07, 35.48it/s]
Loading 0: 26%|██▋ | 96/363 [00:08<00:08, 32.62it/s]
Loading 0: 29%|██▊ | 104/363 [00:08<00:06, 40.73it/s]
Loading 0: 30%|███ | 109/363 [00:08<00:05, 42.54it/s]
Loading 0: 31%|███▏ | 114/363 [00:09<00:06, 36.91it/s]
Loading 0: 33%|███▎ | 121/363 [00:09<00:05, 43.77it/s]
Loading 0: 35%|███▍ | 126/363 [00:09<00:05, 44.17it/s]
Loading 0: 36%|███▌ | 131/363 [00:09<00:05, 44.80it/s]
Loading 0: 38%|███▊ | 137/363 [00:09<00:05, 42.05it/s]
Loading 0: 39%|███▉ | 142/363 [00:09<00:05, 41.60it/s]
Loading 0: 41%|████ | 149/363 [00:09<00:04, 46.65it/s]
Loading 0: 43%|████▎ | 155/363 [00:09<00:04, 45.38it/s]
Loading 0: 44%|████▍ | 160/363 [00:10<00:04, 43.65it/s]
Loading 0: 46%|████▌ | 166/363 [00:10<00:04, 47.29it/s]
Loading 0: 47%|████▋ | 171/363 [00:10<00:04, 46.29it/s]
Loading 0: 48%|████▊ | 176/363 [00:10<00:04, 46.21it/s]
Loading 0: 50%|████▉ | 181/363 [00:10<00:03, 46.93it/s]
Loading 0: 51%|█████ | 186/363 [00:10<00:07, 25.15it/s]
Loading 0: 53%|█████▎ | 193/363 [00:11<00:05, 32.78it/s]
Loading 0: 55%|█████▍ | 198/363 [00:11<00:04, 34.64it/s]
Loading 0: 56%|█████▌ | 203/363 [00:11<00:04, 37.49it/s]
Loading 0: 58%|█████▊ | 209/363 [00:11<00:04, 38.26it/s]
Loading 0: 59%|█████▉ | 214/363 [00:11<00:03, 37.94it/s]
Loading 0: 61%|██████ | 220/363 [00:11<00:03, 42.52it/s]
Loading 0: 62%|██████▏ | 225/363 [00:11<00:03, 42.28it/s]
Loading 0: 63%|██████▎ | 230/363 [00:11<00:03, 42.83it/s]
Loading 0: 65%|██████▍ | 235/363 [00:11<00:02, 43.49it/s]
Loading 0: 66%|██████▌ | 240/363 [00:12<00:03, 36.50it/s]
Loading 0: 68%|██████▊ | 248/363 [00:12<00:02, 44.81it/s]
Loading 0: 70%|██████▉ | 254/363 [00:12<00:02, 43.41it/s]
Loading 0: 71%|███████▏ | 259/363 [00:12<00:02, 41.20it/s]
Loading 0: 73%|███████▎ | 265/363 [00:12<00:02, 45.54it/s]
Loading 0: 74%|███████▍ | 270/363 [00:12<00:02, 45.58it/s]
Loading 0: 76%|███████▌ | 275/363 [00:12<00:01, 44.95it/s]
Loading 0: 77%|███████▋ | 281/363 [00:13<00:01, 42.25it/s]
Loading 0: 79%|███████▉ | 286/363 [00:13<00:01, 41.85it/s]
Loading 0: 81%|████████ | 293/363 [00:13<00:01, 46.75it/s]
Loading 0: 82%|████████▏ | 298/363 [00:13<00:01, 47.37it/s]
Loading 0: 83%|████████▎ | 303/363 [00:13<00:02, 26.31it/s]
Loading 0: 85%|████████▌ | 310/363 [00:13<00:01, 33.33it/s]
Loading 0: 87%|████████▋ | 315/363 [00:14<00:01, 35.65it/s]
Loading 0: 88%|████████▊ | 320/363 [00:14<00:01, 37.87it/s]
Loading 0: 90%|████████▉ | 325/363 [00:14<00:00, 40.32it/s]
Loading 0: 91%|█████████ | 330/363 [00:14<00:00, 34.55it/s]
Loading 0: 93%|█████████▎| 338/363 [00:14<00:00, 42.68it/s]
Loading 0: 95%|█████████▍| 344/363 [00:14<00:00, 41.74it/s]
Loading 0: 96%|█████████▌| 349/363 [00:14<00:00, 40.23it/s]
Loading 0: 98%|█████████▊| 356/363 [00:14<00:00, 45.04it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 43.62it/s]
Job chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer completed after 114.14s with status: succeeded
Stopping job with name chaiml-sao10k-mn-12b-lyra-v4-v1-mkmlizer
Pipeline stage MKMLizer completed in 114.73s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.23s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-sao10k-mn-12b-lyra-v4-v1
Waiting for inference service chaiml-sao10k-mn-12b-lyra-v4-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service chaiml-sao10k-mn-12b-lyra-v4-v1 ready after 120.4704339504242s
Pipeline stage MKMLDeployer completed in 121.08s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9207377433776855s
Received healthy response to inference request in 1.5734760761260986s
Received healthy response to inference request in 1.512943983078003s
Received healthy response to inference request in 1.305647850036621s
Received healthy response to inference request in 1.1504545211791992s
5 requests
0 failed requests
5th percentile: 1.1814931869506835
10th percentile: 1.212531852722168
20th percentile: 1.2746091842651368
30th percentile: 1.3471070766448974
40th percentile: 1.4300255298614502
50th percentile: 1.512943983078003
60th percentile: 1.5371568202972412
70th percentile: 1.5613696575164795
80th percentile: 1.642928409576416
90th percentile: 1.7818330764770507
95th percentile: 1.8512854099273681
99th percentile: 1.906847276687622
mean time: 1.4926520347595216
Pipeline stage StressChecker completed in 9.31s
Shutdown handler de-registered
chaiml-sao10k-mn-12b-lyra-v4_v1 status is now deployed due to DeploymentManager action
chaiml-sao10k-mn-12b-lyra-v4_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-sao10k-mn-12b-lyra-v4_v1 status is now deployed due to admin request
chaiml-sao10k-mn-12b-lyra-v4_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f19b121d-1a99-4dff-9f63-538ed3d64dca, ab3198cc-b7de-4ab4-a1f4-9b5fcf66a458\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:15:53 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7722b5d1-6536-458b-8d60-647139347747, abbaba37-485f-4661-afd4-74f6720d23d9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:41 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'53383b70-1786-482c-8536-343e6119a860, 0638b8be-3000-4f37-9234-1ce9cb290c1e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5654966c-5a7f-4acc-a383-31ff3d3b3ec3, a07aa67e-203f-4f51-be43-4de19fb77e14\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:46 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1a18b81e-971c-4675-95b4-78873ef8720b, e093a1de-d7f8-4d8b-b4b7-76b63e8b6cdc\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:35:46 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5965faab-d15e-41d1-8391-113a76a742e1, 8b8fb896-5794-45c5-afda-bf755a3aa8d7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:40:48 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'83e46621-6767-4e81-b06d-49f38de6790c, ac394651-8f60-408b-b0e7-2775b4c6eded\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:45:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'feada0f5-e3d8-41dd-91fd-2413f8d73af4, a2645ecf-e790-4e5c-98cb-a0a9ae2baff3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:50:48 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5cc79ba1-73be-45db-889e-b32593ac1fa6, 31f91d8a-71e4-4145-833e-6bc46b4e18de\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:55:48 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9e644e6c-3b95-42c3-9414-8612ac35cad9, 74a32de3-3143-427e-86a2-b52787483768\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8d695f27-318c-477a-bafa-85d7963ab16a, a1d822bf-56e4-444c-8eb9-2e686d60e0e3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:05:52 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'acc2d55b-a522-49a1-9c92-6e53cfad1a3f, fec5fa8e-de19-4b51-8284-6f5d32d046de\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:10:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f1703bbf-b734-49bf-9a15-53a81153053b, 57479e00-9439-428a-8de1-2ad9672a1155\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:15:53 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'90174210-8eda-4a1b-bfa6-9ec8ad221ffe, 037c66ef-03bb-4dfd-ad59-aa695cdaec63\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:20:53 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0676c456-f70a-43d4-a4b4-79adeb5c0bcb, 7674f688-d4ad-4a0a-86df-e856fac5fafe\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:25:53 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'204cfbc8-dc32-43f6-b94a-9c1460350c8b, d1639319-c574-43e7-b708-752668d512d2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:30:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'015ba0f7-a3f3-4cd4-851e-c97fb0112060, f1de5bc4-5804-4a9e-94d8-8074d205ef33\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:35:54 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'00e721ac-eb8d-49d1-97b2-644ecf70e038, cc59aa58-f5ec-4ca2-8e8f-9bdba274c6e4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:40:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9739e320-6726-42ce-95c6-61152ab4a315, b9178927-7a38-44a3-8c8e-d0632afe8bd6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:45:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
Checking if service chaiml-rica40325-mistral-8799-v2 is running
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
%s, retrying in %s seconds...
Shutdown handler not registered because Python interpreter is not running in the main thread
Checking if service chaiml-rica40325-mistral-8799-v2 is running
run pipeline %s
run pipeline stage %s
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'28cf93ed-da23-4bca-a6c7-1c062da54a61, 17a89c53-d1f0-4349-b58a-20c8ee0db102\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:50:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Running pipeline stage MKMLDeleter
Shutdown handler de-registered
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7ad8f4e1-cf81-4f5d-8961-647c280c6cbb, 60decd0c-095f-473f-9f74-4b42ec21ba84\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0f627f5b-04f9-4433-948d-bf7046212b65, e460d163-ea0d-428e-ab94-aa741a703ac5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'758c17ba-0940-4720-8fdf-05c5ec520417, b1336793-cdf8-49dd-a084-533c77e3dd7d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f4ff6a67-4c01-459e-86a9-9cb2374455f5, d261e82c-d16b-4d9a-b43a-4645246bbb10\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9a50410d-bcc3-4327-8cc8-306dfd6f7225, bb49c704-9d24-4bc7-9349-16602b07c5fb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:02 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d8887978-4a70-4697-b22f-3139786e05aa, be99f1b7-c1dc-49ee-b5bb-20fd67e4f14a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'76f16299-86ab-4f71-b486-a883d2cf91d1, 3fd3d845-6b6a-4bbf-90b2-6fb9488eec6c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-sao10k-mn-12b-lyra-v4_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-sao10k-mn-12b-lyra-v4-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'72841729-33d2-4f84-91a4-e5a561726c6e, 7758bf80-eccd-4c90-959b-c8cb20e04df9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
chaiml-sao10k-mn-12b-lyra-v4_v1 status is now torndown due to DeploymentManager action