nvidia-mistral-nemo-mini_3107

submission_id: nvidia-mistral-nemo-mini_3107_v1
developer_uid: chai_backend_admin
best_of: 8
celo_rating: 1224.2
display_name: nvidia-mistral-nemo-mini_3107_v1
family_friendly_score: 0.556467723810713
family_friendly_standard_error: 0.00551775693422737
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
is_internal_developer: True
language_model: nvidia/Mistral-NeMo-Minitron-8B-Instruct
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: nvidia/Mistral-NeMo-Mini
model_name: nvidia-mistral-nemo-mini_3107_v1
model_num_parameters: 8414105600.0
model_repo: nvidia/Mistral-NeMo-Minitron-8B-Instruct
model_size: 8B
num_battles: 8585
num_wins: 3979
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-10-05T04:04:46+00:00
us_pacific_date: 2024-10-04
win_ratio: 0.4634828188701223
Download Preference Data
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nvidia-mistral-nemo-mini-3107-v1-mkmlizer
Waiting for job on nvidia-mistral-nemo-mini-3107-v1-mkmlizer to finish
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║     _____            __           __                                ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║       /___/                                                         ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║                                                                     ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  https://mk1.ai                                                     ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║                                                                     ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  belonging to:                                                      ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║                                                                     ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  Chai Research Corp.                                                ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ║                                                                     ║
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-elo-alignment-run-3_v44: ('http://chaiml-elo-alignment-run-3-v44-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:51942->127.0.0.1:8080: read: connection reset by peer\n')
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: Downloaded to shared memory in 38.103s
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpq2vpzxj1, device:0
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: quantized model in 31.665s
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: Processed model nvidia/Mistral-NeMo-Minitron-8B-Instruct in 69.768s
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: creating bucket guanaco-mkml-models
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nvidia-mistral-nemo-mini-3107-v1
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nvidia-mistral-nemo-mini-3107-v1/config.json
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nvidia-mistral-nemo-mini-3107-v1/special_tokens_map.json
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nvidia-mistral-nemo-mini-3107-v1/tokenizer_config.json
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nvidia-mistral-nemo-mini-3107-v1/tokenizer.json
nvidia-mistral-nemo-mini-3107-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nvidia-mistral-nemo-mini-3107-v1/flywheel_model.0.safetensors
Job nvidia-mistral-nemo-mini-3107-v1-mkmlizer completed after 93.85s with status: succeeded
Stopping job with name nvidia-mistral-nemo-mini-3107-v1-mkmlizer
Pipeline stage MKMLizer completed in 94.38s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service nvidia-mistral-nemo-mini-3107-v1
Waiting for inference service nvidia-mistral-nemo-mini-3107-v1 to be ready
Inference service nvidia-mistral-nemo-mini-3107-v1 ready after 120.43024969100952s
Pipeline stage MKMLDeployer completed in 120.98s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9197423458099365s
Received healthy response to inference request in 1.7410790920257568s
Received healthy response to inference request in 1.2880885601043701s
Received healthy response to inference request in 1.481642723083496s
Received healthy response to inference request in 1.2605559825897217s
5 requests
0 failed requests
5th percentile: 1.2660624980926514
10th percentile: 1.271569013595581
20th percentile: 1.2825820446014404
30th percentile: 1.3267993927001953
40th percentile: 1.4042210578918457
50th percentile: 1.481642723083496
60th percentile: 1.5854172706604004
70th percentile: 1.6891918182373047
80th percentile: 1.776811742782593
90th percentile: 1.8482770442962646
95th percentile: 1.8840096950531005
99th percentile: 1.9125958156585694
mean time: 1.5382217407226562
Pipeline stage StressChecker completed in 9.41s
Shutdown handler de-registered
nvidia-mistral-nemo-mini_3107_v1 status is now deployed due to DeploymentManager action
nvidia-mistral-nemo-mini_3107_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4b440e79-9fb2-4cf2-ad70-8faa3eab437e, 73d741fb-c1a6-47b4-895c-5a29c1858b85\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'893b1f3e-9396-40e6-9153-682ab56e46e1, ae8bce8d-b637-470e-aff9-74a978d0a691\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Checking if service nousresearch-meta-llama-4939-v69 is running
Shutdown handler not registered because Python interpreter is not running in the main thread
%s, retrying in %s seconds...
run pipeline %s
Checking if service nousresearch-meta-llama-4939-v69 is running
run pipeline stage %s
%s, retrying in %s seconds...
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c35db8f0-ff4e-445d-a9ab-108f9998ffaf, 95ae5b43-0ab6-415a-91e4-d49b32f533fa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
%s, retrying in %s seconds...
Shutdown handler de-registered
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7f2d5eed-3163-49e8-bbf6-813083b29f97, d45073de-ca08-46b4-abd1-cc43cb6f46b1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'97031220-1d40-4a6c-8311-f6027c5aee69, 3bac45cf-f186-4253-9817-9c3658a61fe6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'73feff3a-757c-41ce-a213-420b0604071f, 09f083cb-409c-44f6-b15b-3cf79625571d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:36:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'41a5f870-78ab-4349-bcf9-582a452b5c69, c0fc10ba-eeff-4b40-ae02-68e116a49748\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ad611ebb-122e-4ed1-8442-98ea7c163bb8, 87c96aa0-5189-4ffc-8c32-623797a8184d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'78e5f10e-b7dc-4fdf-8479-24ea43ae810f, 798bd9f5-d168-408b-8a56-2ad7bdf3dca9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'539c657b-3b1a-42a8-8c4d-5bb6a5c05984, 6716c9b0-ac66-4540-b1e3-b40419bc546c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4acbcb50-5a8f-4589-aad3-472eb4af325c, 527c52bd-d349-4a99-8cbc-3ec1d32905e6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'02c9be1e-9099-4945-96ff-ad017e8dfba0, 41b389fc-036d-4a5f-86b8-6af491d41576\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c0ab9295-42b7-4094-8010-be7a3e04bb53, b89d1746-de55-4c51-b506-b275141d44ab\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3ad767fe-c23e-4ed6-a54f-1f119f60918a, 1104ab55-8e4f-4d57-90a9-180b6b9d435e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:18 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e9044b2f-b650-4468-bfa7-d556598831b0, 35e57dab-358f-4f73-a221-cf6d986021e2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'709a4437-c3f4-41ec-be58-31b352667ac3, 2ac6256a-3561-4cb3-b665-2d9781e807e7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'90135608-f4a4-4f01-84d9-018c62e2fcb0, d1ae2122-2d8f-4e37-89df-d5c3a442a4eb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:22 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fc2dc90e-7d4e-4755-b787-812a68f52a14, 1d2cc74a-105a-4740-9ee5-4f2651c5c10b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'90868615-73e5-48c3-8cde-29b60ce451e4, 0369dae9-2571-4dfc-b8be-f1a90f26b92f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:25 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'48ab2e0f-b417-497c-994d-e252a6ce5dce, 037f6830-3c42-4b68-bb86-547308254707\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'95fa2014-1a7e-4cd5-a215-a3ef8cb6ed46, 03afbcc9-1a96-49f5-bce7-fa948dcf0c4d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'43b343fa-7d63-4dc5-b717-e00d2e7ee714, f89ba5f8-0564-405e-965e-d1dc0e6c6254\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dd3f2ed8-2d79-4207-8130-f40ada4b99a3, 20fe8bfa-e2a0-479c-aef9-378c81bb51e6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:41 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9325186e-6a82-4569-a142-754e7f84eb9a, 4ae740f3-816c-46ba-833f-fcbd34a0b377\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'883a4869-1301-4ce7-a9f0-ed5dd1125b3c, f296d8ae-4371-44b3-9dad-ccd6980ee5a0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:31 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0a9244c1-b742-43a5-95b3-f2bcd1e9f7f9, 5b216574-499a-442a-a083-bedaa1a2e4cf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:31 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'aabbbab7-b307-4f36-a65e-8a725c5a7c14, 004ceacc-dbc0-467c-96e7-16fa0d2c8fa2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:34 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
%s, retrying in %s seconds...
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0206100d-1de3-4d35-a61d-1282bddd2253, b038a3f8-1bdc-4d45-a5c6-7c070187735f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:31 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
nvidia-mistral-nemo-mini_3107_v1 status is now torndown due to DeploymentManager action