jic062-dpo-v2-3-nemo

developer_uid: chace9580
submission_id: jic062-dpo-v2-3-nemo_v1
model_name: jic062-dpo-v2-3-nemo_v1
model_group: jic062/dpo-v2.3-Nemo
status: torndown
timestamp: 2024-10-03T21:06:21+00:00
num_battles: 3642
num_wins: 1911
celo_rating: 1268.82
family_friendly_score: 0.5677966101694916
family_friendly_standard_error: 0.008312832385006077
submission_type: basic
model_repo: jic062/dpo-v2.3-Nemo
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: jic062-dpo-v2-3-nemo_v1
ineligible_reason: num_battles<5000
is_internal_developer: False
language_model: jic062/dpo-v2.3-Nemo
model_size: 13B
ranking_group: single
us_pacific_date: 2024-10-03
win_ratio: 0.5247116968698518
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '/s', '[/INST]'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '[INST]system\n{memory}[/INST]\n', 'prompt_template': '[INST]user\n{prompt}[/INST]\n', 'bot_template': '[INST]assistant\n{bot_name}: {message}[/INST]\n', 'user_template': '[INST]user\n{user_name}: {message}[/INST]\n', 'response_template': '[INST]assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-dpo-v2-3-nemo-v1-mkmlizer
Waiting for job on jic062-dpo-v2-3-nemo-v1-mkmlizer to finish
jic062-dpo-v2-3-nemo-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║     _____            __           __                                ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║       /___/                                                         ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║                                                                     ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  https://mk1.ai                                                     ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║                                                                     ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  belonging to:                                                      ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║                                                                     ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  Chai Research Corp.                                                ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ║                                                                     ║
jic062-dpo-v2-3-nemo-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
jic062-dpo-v2-3-nemo-v1-mkmlizer: quantized model in 38.801s
jic062-dpo-v2-3-nemo-v1-mkmlizer: Processed model jic062/dpo-v2.3-Nemo in 88.758s
jic062-dpo-v2-3-nemo-v1-mkmlizer: creating bucket guanaco-mkml-models
jic062-dpo-v2-3-nemo-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-dpo-v2-3-nemo-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-dpo-v2-3-nemo-v1
jic062-dpo-v2-3-nemo-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-dpo-v2-3-nemo-v1/config.json
jic062-dpo-v2-3-nemo-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-dpo-v2-3-nemo-v1/special_tokens_map.json
jic062-dpo-v2-3-nemo-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-dpo-v2-3-nemo-v1/tokenizer_config.json
jic062-dpo-v2-3-nemo-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-dpo-v2-3-nemo-v1/tokenizer.json
Job jic062-dpo-v2-3-nemo-v1-mkmlizer completed after 202.47s with status: succeeded
Stopping job with name jic062-dpo-v2-3-nemo-v1-mkmlizer
admin requested tearing down of blend_rofur_2024-10-03
Pipeline stage MKMLizer completed in 222.04s
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline stage %s
run pipeline %s
Tearing down inference service blend-rofur-2024-10-03
Running pipeline stage MKMLTemplater
admin requested tearing down of blend_rofur_2024-10-03
run pipeline stage %s
%s, retrying in %s seconds...
Tearing down inference service blend-rofur-2024-10-03
Pipeline stage MKMLTemplater completed in 11.40s
Shutdown handler not registered because Python interpreter is not running in the main thread
Tearing down inference service blend-rofur-2024-10-03
Tearing down inference service blend-rofur-2024-10-03
Running pipeline stage ProductionBlendMKMLTemplater
Creating inference service blend-rofur-2024-10-03
%s, retrying in %s seconds...
run pipeline stage %s
run pipeline %s
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Pipeline stage %s skipped, reason=%s
Waiting for inference service blend-rofur-2024-10-03 to be ready
Creating inference service blend-rofur-2024-10-03
Running pipeline stage MKMLDeployer
run pipeline stage %s
Creating inference service blend-rofur-2024-10-03
Creating inference service blend-rofur-2024-10-03
Pipeline stage ProductionBlendMKMLTemplater completed in 33.00s
Ignoring service blend-rofur-2024-10-03 already deployed
Creating inference service jic062-dpo-v2-3-nemo-v1
Running pipeline stage ProductionBlendMKMLTemplater
Ignoring service blend-rofur-2024-10-03 already deployed
Ignoring service blend-rofur-2024-10-03 already deployed
run pipeline stage %s
Waiting for inference service blend-rofur-2024-10-03 to be ready
Waiting for inference service jic062-dpo-v2-3-nemo-v1 to be ready
Pipeline stage %s skipped, reason=%s
Waiting for inference service blend-rofur-2024-10-03 to be ready
Waiting for inference service blend-rofur-2024-10-03 to be ready
Running pipeline stage MKMLDeployer
Pipeline stage ProductionBlendMKMLTemplater completed in 30.71s
Creating inference service blend-rofur-2024-10-03
run pipeline stage %s
Ignoring service blend-rofur-2024-10-03 already deployed
Running pipeline stage MKMLDeployer
Waiting for inference service blend-rofur-2024-10-03 to be ready
Creating inference service blend-rofur-2024-10-03
Ignoring service blend-rofur-2024-10-03 already deployed
Waiting for inference service blend-rofur-2024-10-03 to be ready
Inference service jic062-dpo-v2-3-nemo-v1 ready after 110.47287631034851s
Pipeline stage MKMLDeployer completed in 163.30s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2606289386749268s
Received healthy response to inference request in 2.0269312858581543s
Received healthy response to inference request in 2.3241076469421387s
Received healthy response to inference request in 2.0305063724517822s
Received healthy response to inference request in 2.1539101600646973s
5 requests
0 failed requests
5th percentile: 2.02764630317688
10th percentile: 2.0283613204956055
20th percentile: 2.0297913551330566
30th percentile: 2.055187129974365
40th percentile: 2.1045486450195314
50th percentile: 2.1539101600646973
admin requested tearing down of blend_rofur_2024-10-03
60th percentile: 2.196597671508789
Shutdown handler not registered because Python interpreter is not running in the main thread
70th percentile: 2.239285182952881
run pipeline %s
admin requested tearing down of blend_rofur_2024-10-03
80th percentile: 2.2733246803283693
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
90th percentile: 2.298716163635254
Running pipeline stage ProductionBlendMKMLTemplater
run pipeline %s
95th percentile: 2.311411905288696
Pipeline stage %s skipped, reason=%s
run pipeline stage %s
99th percentile: 2.32156849861145
Pipeline stage ProductionBlendMKMLTemplater completed in 21.12s
Running pipeline stage ProductionBlendMKMLTemplater
mean time: 2.15921688079834
run pipeline stage %s
Pipeline stage %s skipped, reason=%s
Pipeline stage StressChecker completed in 148.77s
Running pipeline stage MKMLDeployer
Pipeline stage ProductionBlendMKMLTemplater completed in 20.72s
run pipeline stage %s
Creating inference service blend-rofur-2024-10-03
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
Ignoring service blend-rofur-2024-10-03 already deployed
Running pipeline stage MKMLDeployer
run_pipeline:run_in_cloud %s
Waiting for inference service blend-rofur-2024-10-03 to be ready
Creating inference service blend-rofur-2024-10-03
starting trigger_guanaco_pipeline args=%s
Ignoring service blend-rofur-2024-10-03 already deployed
Pipeline stage TriggerMKMLProfilingPipeline completed in 34.26s
Waiting for inference service blend-rofur-2024-10-03 to be ready
Shutdown handler de-registered
jic062-dpo-v2-3-nemo_v1 status is now deployed due to DeploymentManager action
jic062-dpo-v2-3-nemo_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'204184e5-5fef-4514-bafe-b3d8b8eac7b9, 767d04dd-c4f3-4864-90c3-789fad3d65c5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bbbac99e-12bb-4b65-b6ff-b094ddba34e2, 636e5e70-0bb4-4371-b275-2e37be5995dd\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'17014db2-09bc-4893-b08b-42cb2a582a60, 2e2bb2e1-9200-4742-ac7e-5f40d98a8f47\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'055888e3-315b-461a-b56c-fd892e03f973, 921d7dec-4a47-431d-b05c-ace2cb12db5c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:54 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ea4696e3-0ef7-4c33-9f0e-c8a27c67db92, 7c3ec1f7-c457-4226-ba02-34706d31d225\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:35:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1d4d390b-27db-44c4-92c4-b69a37709faa, 1db7bf1e-3bd6-4f4d-8af6-1e700dfc5365\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:40:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cc48f906-4d67-4e80-9611-d3bfbacbb037, 5f18bbed-e61e-4558-addc-ababeed71bca\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:45:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c1a88f83-91b0-4872-a401-55bc973ca2d9, f2a34edd-2187-4847-8403-d62041cdd0e8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:50:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f0cd27ff-6fb1-43f7-9305-c84a28fcdf5c, 2d487c27-4627-4204-90e4-4144916b886e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:55:57 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c9a52341-22cd-473a-b855-d4926d25f4cc, fd408b21-a3d9-4cb2-8ca5-cb218be4d14c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'213ff2c4-df47-4ee9-bbc4-ec366df19ba8, 195fa2e6-ebe2-40c6-9cca-fbfda160468f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:02 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'02eb4ac2-c1a3-4e6c-9ef7-4285ec5fd76e, 4aeab7b3-10e2-49bc-b6c3-3cd934a212d7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1765df1c-f863-4259-9ea9-3ba6c72db407, ae5b0b3e-1b08-486b-8a57-589562c8ac24\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4c954ff6-7370-4d83-90a7-0950c5e71305, 5cb569fd-3a2d-43ed-ae7b-9a2b6ca65e59\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9aa393b9-36b7-41b7-a132-bf76144428ba, b40df917-205c-42c8-8a90-3461ebfbae62\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'897a7f1a-8301-459a-a9b9-551d2e208b3d, db0f9e97-17a1-4a1f-aaa4-1bbf7978b12a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a6c30c92-0204-4ce4-a185-b0b1c8eca4ab, 4339a0ea-2f9d-4f47-8925-56fbeace9876\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6e066286-f07a-4621-a2b8-38e06d505d0a, ff475837-24d2-45b1-8475-f482cb3534ed\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dbc53805-38e0-41d1-be1a-f9bd9f8a1428, 5072eeb1-62e6-44b7-9e54-3687315459e0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2b2c1572-247c-4568-b432-8ab852ccd46c, 742be15b-b31a-4f42-9182-497abe165430\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bbca06a9-11bd-46d0-9576-018748abca7a, f39834d1-04c6-472f-a5bc-1dab3309a370\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'89289a69-e2f7-47c4-b7cc-8c1bd39e2efb, 18a19f2a-3b0c-437b-905f-53b17a6f79a0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:25 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'547015f7-15c5-4574-b2da-a4b5af4b0cc4, 3883d755-4b51-4160-b732-7dc58720750f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bd4c91ce-baf0-4106-9ae5-b1d2eba86c1f, ef3cfe0a-c151-47b4-8bb6-ae1b8412af83\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8deb7a2f-9804-430c-8604-799c94605afa, 54801589-97bb-485b-8fde-4f74b9485c0f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c3db8e63-650c-402b-9d3e-c548a6357b58, 2339f581-4a74-4d93-a88a-fe8967b584c7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-dpo-v2-3-nemo_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-dpo-v2-3-nemo-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'69348e83-adff-4fcf-9812-b49e9cdc5d94, db3afb1a-5cf1-43c8-b65c-67d20a031f09\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
jic062-dpo-v2-3-nemo_v1 status is now torndown due to DeploymentManager action