Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-viral-ss-v0-1e5-v4-mkmlizer
Waiting for job on zonemercy-viral-ss-v0-1e5-v4-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-viral-ss-v0-1e5-v5-mkmlizer
Waiting for job on zonemercy-viral-ss-v0-1e5-v5-mkmlizer to finish
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ _____ __ __ ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ /___/ ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ Version: 0.11.12 ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ https://mk1.ai ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ The license key for the current software has been verified as ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ _____ __ __ ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ belonging to: ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ Chai Research Corp. ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ /___/ ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ Version: 0.11.12 ║
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ https://mk1.ai ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ The license key for the current software has been verified as ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ belonging to: ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ Chai Research Corp. ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: Downloaded to shared memory in 56.953s
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: Downloaded to shared memory in 53.272s
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpfuausq92, device:0
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp68z54kt3, device:0
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: quantized model in 50.286s
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: quantized model in 49.275s
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: Processed model zonemercy/Viral-ss-v0-1e5 in 107.240s
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: Processed model zonemercy/Viral-ss-v0-1e5 in 102.547s
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v4
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v5
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v4/config.json
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v5/special_tokens_map.json
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v4/special_tokens_map.json
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v5/config.json
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v4/tokenizer_config.json
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v5/tokenizer_config.json
zonemercy-viral-ss-v0-1e5-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v4/tokenizer.json
zonemercy-viral-ss-v0-1e5-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5-v5/tokenizer.json
Job zonemercy-viral-ss-v0-1e5-v4-mkmlizer completed after 185.76s with status: succeeded
Stopping job with name zonemercy-viral-ss-v0-1e5-v4-mkmlizer
Job zonemercy-viral-ss-v0-1e5-v5-mkmlizer completed after 185.99s with status: succeeded
Pipeline stage MKMLizer completed in 188.97s
Stopping job with name zonemercy-viral-ss-v0-1e5-v5-mkmlizer
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLizer completed in 191.71s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 1.62s
run pipeline stage %s
Pipeline stage MKMLTemplater completed in 0.13s
Running pipeline stage MKMLDeployer
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-viral-ss-v0-1e5-v4
Creating inference service zonemercy-viral-ss-v0-1e5-v5
Waiting for inference service zonemercy-viral-ss-v0-1e5-v4 to be ready
Waiting for inference service zonemercy-viral-ss-v0-1e5-v5 to be ready
Inference service zonemercy-viral-ss-v0-1e5-v4 ready after 120.49073529243469s
Inference service zonemercy-viral-ss-v0-1e5-v5 ready after 120.47610974311829s
Pipeline stage MKMLDeployer completed in 123.02s
Pipeline stage MKMLDeployer completed in 123.21s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage StressChecker
Running pipeline stage StressChecker
Received healthy response to inference request in 5.006299734115601s
Received healthy response to inference request in 5.537997007369995s
Received healthy response to inference request in 3.6055376529693604s
Received healthy response to inference request in 8.090059995651245s
Received healthy response to inference request in 6.658735036849976s
Received healthy response to inference request in 5.613718748092651s
Received healthy response to inference request in 5.7993574142456055s
Received healthy response to inference request in 3.179882526397705s
5 requests
0 failed requests
5th percentile: 3.265013551712036
10th percentile: 3.3501445770263674
20th percentile: 3.520406627655029
30th percentile: 3.8856900691986085
40th percentile: 4.445994901657104
50th percentile: 5.006299734115601
60th percentile: 5.3235228061676025
70th percentile: 5.6407458782196045
80th percentile: 5.971232938766479
90th percentile: 6.314983987808228
Received healthy response to inference request in 8.282075643539429s
95th percentile: 6.486859512329102
99th percentile: 6.624359931945801
mean time: 4.849962472915649
%s, retrying in %s seconds...
Received healthy response to inference request in 2.801835536956787s
5 requests
0 failed requests
5th percentile: 3.3490678310394286
10th percentile: 3.89630012512207
20th percentile: 4.990764713287353
30th percentile: 5.553141355514526
40th percentile: 5.583430051803589
50th percentile: 5.613718748092651
60th percentile: 6.604255247116089
70th percentile: 7.594791746139526
80th percentile: 8.128463125228881
90th percentile: 8.205269384384156
95th percentile: 8.243672513961792
99th percentile: 8.2743950176239
mean time: 6.0651373863220215
%s, retrying in %s seconds...
Received healthy response to inference request in 2.7879419326782227s
Received healthy response to inference request in 5.237815618515015s
Received healthy response to inference request in 3.9347269535064697s
Received healthy response to inference request in 3.739818811416626s
Received healthy response to inference request in 2.6111984252929688s
Received healthy response to inference request in 1.7862873077392578s
Received healthy response to inference request in 6.987743377685547s
Received healthy response to inference request in 6.581279993057251s
{"detail":"module 'guanaco_model_service.chat_api' has no attribute 'get_default_reward'"}
Received unhealthy response to inference request!
5 requests
1 failed requests
5th percentile: 0.5749897003173828
10th percentile: 0.8778141021728516
20th percentile: 1.4834629058837892
30th percentile: 2.1769936084747314
Received healthy response to inference request in 4.312154054641724s
40th percentile: 2.9584062099456787
5 requests
50th percentile: 3.739818811416626
0 failed requests
60th percentile: 4.339017534255982
5th percentile: 2.6465471267700194
70th percentile: 4.938216257095337
10th percentile: 2.6818958282470704
80th percentile: 5.506508493423462
20th percentile: 2.752593231201172
90th percentile: 6.043894243240357
30th percentile: 3.017298936843872
95th percentile: 6.312587118148803
40th percentile: 3.476012945175171
99th percentile: 6.527541418075561
50th percentile: 3.9347269535064697
mean time: 3.5234734058380126
60th percentile: 4.085697793960572
%s, retrying in %s seconds...
70th percentile: 4.236668634414673
80th percentile: 4.847271919250488
90th percentile: 5.917507648468018
Received healthy response to inference request in 2.5551798343658447s
95th percentile: 6.452625513076782
99th percentile: 6.880719804763794
mean time: 4.126752948760986
%s, retrying in %s seconds...
Received healthy response to inference request in 2.4919936656951904s
Received healthy response to inference request in 2.4193270206451416s
Received healthy response to inference request in 2.3088722229003906s
Received healthy response to inference request in 2.5296313762664795s
Received healthy response to inference request in 2.6812872886657715s
Received healthy response to inference request in 2.74259352684021s
Received healthy response to inference request in 3.145118474960327s
5 requests
0 failed requests
5th percentile: 2.3454965114593507
10th percentile: 2.3821208000183107
20th percentile: 2.4553693771362304
30th percentile: 2.4995212078094484
40th percentile: 2.514576292037964
50th percentile: 2.5296313762664795
60th percentile: 2.5398507595062254
70th percentile: 2.5500701427459718
80th percentile: 2.6731675624847413
90th percentile: 2.909143018722534
95th percentile: 3.0271307468414306
99th percentile: 3.121520929336548
mean time: 2.6061591148376464
Pipeline stage StressChecker completed in 91.92s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.5351996421813965s
Received healthy response to inference request in 3.0058932304382324s
5 requests
Pipeline stage TriggerMKMLProfilingPipeline completed in 4.36s
0 failed requests
Shutdown handler de-registered
5th percentile: 2.4425015449523926
zonemercy-viral-ss-v0-1e5_v5 status is now deployed due to DeploymentManager action
10th percentile: 2.4656760692596436
20th percentile: 2.5120251178741455
30th percentile: 2.5644171714782713
40th percentile: 2.6228522300720214
50th percentile: 2.6812872886657715
60th percentile: 2.705809783935547
70th percentile: 2.730332279205322
80th percentile: 2.7952534675598146
90th percentile: 2.9005733489990235
95th percentile: 2.9532332897186278
99th percentile: 2.9953612422943117
mean time: 2.67686014175415
Pipeline stage StressChecker completed in 97.43s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.45s
Shutdown handler de-registered
zonemercy-viral-ss-v0-1e5_v4 status is now deployed due to DeploymentManager action
zonemercy-viral-ss-v0-1e5_v4 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e63bea6c-3a22-476c-b742-ab9d41ebb0ae, c80a1087-e754-4712-b9fe-cb39734ca70b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e31f6dc7-b091-42d0-8c48-8a53d786a9ca, a0c301f1-6993-42bf-84bc-43b4578629e9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d5bd3869-b180-46df-81df-41036d220d4e, 2274b022-f543-4935-8e7e-3bc8314c4220\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'841576a5-06c1-4b52-bc36-9001f349af82, 091481e4-45df-44ab-a500-2c44acb317cc\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bb141ea9-7dc7-48a9-b44c-20d67aad41a0, 96896f3f-f332-4e7e-aacb-86455334a31f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:38:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'89ccc6fd-8b90-4dad-941b-29422fc97845, 70d38320-0fe2-40ac-8424-d790e4c7af4b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'892c6edd-a57c-4f05-bf2a-414154006818, 6d9e8873-d48b-4b70-bee4-ba02b8fa8daf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:41 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'65717ab6-f651-4e3b-a6f5-d533d8456339, 5385b79d-1b8d-4ca5-a54b-2f78c081c351\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:42 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5b0d3ade-85df-4956-8086-ca48490dff05, 7f49a67c-54a9-45cf-9a0e-ede918d99d2e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:42 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bfa4ced8-4204-4e93-9bc1-b2e45a44c848, ce7519d6-bfc0-4bb1-8f00-01801ab346e9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f17639f5-730e-4fbd-868e-eb718b6f9bdb, daf595a1-928a-4d39-bfa2-387437f1d100\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'af6ae4a0-4007-40e7-8e18-953985d932e4, e8b1fb57-c314-44e6-aadb-971ad8b528a8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4d098b35-155e-471f-a78d-ee344c7039d5, 69a36863-e748-4cf2-9d52-1a9da579f50c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'555f6afa-64b6-4907-a5f1-da473a0375cb, b7f3e847-61cd-401a-82ca-72ef63fc98f9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9d370eb4-c64c-44a8-a6be-4683b12b07e0, 77b76718-9f9b-44b4-b995-4c855c556122\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'367a05be-a48d-4d2e-b7b3-aeea66007df1, ea641130-bcbc-4606-96be-e0e556faa3d4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4559219f-a07d-4901-b14b-cb6f36e948fd, b5ef6d43-6ecb-4317-b60c-399f98ae094f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2f8564f3-9c22-452c-8dd4-1307575e5f93, 535f057e-ffb6-4b47-a0d4-717f38438bc5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:42:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7fc5c294-3c59-4e78-92c8-e9765590c1e2, 796c742f-9073-45a5-8059-88ced7bdc66c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:47:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'29d368ec-530d-4a51-87a9-99c19c30a906, fa21a434-70e8-4262-82da-a5c16d3bc636\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:52:07 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'218e3e30-db06-4068-8df0-c70bc7916364, ff7c80aa-8911-4672-83fa-b382d1ad2791\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:57:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f47b094a-92b6-49d4-a66c-98b7e925ba5d, ab9b0617-1413-4d60-bad5-9798a975c688\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:02:23 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f606e452-0adf-4106-acdd-66b51f25dd9d, cf7c6a62-2f6a-4981-ad37-1178d4dbeca3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:07:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v7
%s, retrying in %s seconds...
Shutdown handler not registered because Python interpreter is not running in the main thread
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
run pipeline %s
run pipeline stage %s
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'015f1be1-83c9-411f-9ee8-8afc3c0f54d1, b09bda94-30f9-4a12-96db-d458f2ff19c1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:12:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
admin requested tearing down of zonemercy-viral-ss-v0-1e5ep2_v15
Running pipeline stage MKMLDeleter
Shutdown handler de-registered
Shutdown handler not registered because Python interpreter is not running in the main thread
Checking if service zonemercy-viral-ss-v0-1e5-v7 is running
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'964add2b-8547-4675-818e-ca6b303ba7d5, 5949738c-7f08-4d27-8c0a-c4d10ff98aba\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:17:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e95c894e-28c3-41d1-adc5-4f5ff85831e7, 1770838d-57dd-432f-9afc-48659ba4d9ea\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:22:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-viral-ss-v0-1e5-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'408f8a30-f10e-42ab-8fdd-d8fb97fa22cc, 6b695334-620a-43ef-bf23-542ab49d2495\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:27:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
zonemercy-viral-ss-v0-1e5_v4 status is now torndown due to DeploymentManager action