mistralai-mistral-nemo_9330

developer_uid: robert_irvine
submission_id: mistralai-mistral-nemo_9330_v139
model_name: mistralai-mistral-nemo_9330_v139
model_group: mistralai/Mistral-Nemo-I
status: torndown
timestamp: 2024-10-04T20:34:36+00:00
num_battles: 3291
num_wins: 1641
celo_rating: 1250.8
family_friendly_score: 0.5703541209652147
family_friendly_standard_error: 0.008753387788283768
submission_type: basic
model_repo: mistralai/Mistral-Nemo-Instruct-2407
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: mistralai-mistral-nemo_9330_v139
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: mistralai/Mistral-Nemo-Instruct-2407
model_size: 13B
ranking_group: single
us_pacific_date: 2024-10-04
win_ratio: 0.4986326344576117
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '###', 'Bot:', 'User:', 'You:', '<|im_end|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nemo-9330-v139-mkmlizer
Waiting for job on mistralai-mistral-nemo-9330-v139-mkmlizer to finish
mistralai-mistral-nemo-9330-v139-mkmlizer: Downloaded to shared memory in 51.683s
mistralai-mistral-nemo-9330-v139-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpwq_wo8x_, device:0
mistralai-mistral-nemo-9330-v139-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nemo-9330-v139-mkmlizer: quantized model in 38.133s
mistralai-mistral-nemo-9330-v139-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 89.817s
mistralai-mistral-nemo-9330-v139-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nemo-9330-v139-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nemo-9330-v139-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v139
mistralai-mistral-nemo-9330-v139-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v139/special_tokens_map.json
mistralai-mistral-nemo-9330-v139-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v139/config.json
mistralai-mistral-nemo-9330-v139-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v139/tokenizer_config.json
mistralai-mistral-nemo-9330-v139-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v139/tokenizer.json
mistralai-mistral-nemo-9330-v139-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v139/flywheel_model.0.safetensors
mistralai-mistral-nemo-9330-v139-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 29.79it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 49.83it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 46.54it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 44.87it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 50.59it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 47.09it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 45.09it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 50.22it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 48.13it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 35.74it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:08, 35.72it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 39.81it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:07, 39.80it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:07, 39.36it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:06, 42.91it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 43.02it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:05, 44.06it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:06, 42.98it/s]
Loading 0:  30%|███       | 110/363 [00:02<00:05, 44.69it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 44.83it/s]
Loading 0:  33%|███▎      | 120/363 [00:02<00:05, 42.06it/s]
Loading 0:  34%|███▍      | 125/363 [00:02<00:05, 43.48it/s]
Loading 0:  36%|███▌      | 130/363 [00:03<00:05, 42.72it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 40.79it/s]
Loading 0:  39%|███▊      | 140/363 [00:03<00:05, 42.33it/s]
Loading 0:  40%|███▉      | 145/363 [00:03<00:08, 26.77it/s]
Loading 0:  41%|████      | 149/363 [00:03<00:07, 27.19it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 35.23it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:05, 37.39it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:05, 38.67it/s]
Loading 0:  47%|████▋     | 171/363 [00:04<00:04, 41.33it/s]
Loading 0:  48%|████▊     | 176/363 [00:04<00:05, 35.36it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:04, 43.74it/s]
Loading 0:  52%|█████▏    | 190/363 [00:04<00:04, 42.64it/s]
Loading 0:  54%|█████▎    | 195/363 [00:04<00:04, 41.27it/s]
Loading 0:  55%|█████▌    | 201/363 [00:04<00:03, 44.20it/s]
Loading 0:  57%|█████▋    | 206/363 [00:05<00:03, 44.12it/s]
Loading 0:  58%|█████▊    | 211/363 [00:05<00:03, 44.26it/s]
Loading 0:  60%|█████▉    | 216/363 [00:05<00:03, 45.67it/s]
Loading 0:  61%|██████    | 222/363 [00:05<00:03, 43.61it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:04, 31.24it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:04, 30.00it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 35.94it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 37.33it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:02, 38.91it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 41.48it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:03, 34.05it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 40.78it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 40.00it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 39.55it/s]
Loading 0:  77%|███████▋  | 279/363 [00:06<00:02, 41.41it/s]
Loading 0:  78%|███████▊  | 284/363 [00:07<00:02, 34.93it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 42.02it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 42.73it/s]
Loading 0:  83%|████████▎ | 302/363 [00:07<00:01, 46.37it/s]
Loading 0:  85%|████████▍ | 307/363 [00:14<00:22,  2.50it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:15,  3.38it/s]
Loading 0:  88%|████████▊ | 319/363 [00:14<00:08,  5.13it/s]
Loading 0:  89%|████████▉ | 324/363 [00:14<00:05,  6.67it/s]
Loading 0:  91%|█████████ | 329/363 [00:14<00:03,  8.59it/s]
Loading 0:  92%|█████████▏| 333/363 [00:15<00:02, 10.48it/s]
Loading 0:  93%|█████████▎| 337/363 [00:15<00:02, 12.60it/s]
Loading 0:  94%|█████████▍| 341/363 [00:15<00:01, 14.65it/s]
Loading 0:  95%|█████████▌| 345/363 [00:15<00:01, 17.72it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 19.21it/s]
Loading 0:  97%|█████████▋| 353/363 [00:15<00:00, 22.09it/s]
Loading 0:  98%|█████████▊| 357/363 [00:15<00:00, 22.74it/s]
Loading 0: 100%|█████████▉| 362/363 [00:16<00:00, 27.51it/s]
                                                            
Job mistralai-mistral-nemo-9330-v139-mkmlizer completed after 143.4s with status: succeeded
Stopping job with name mistralai-mistral-nemo-9330-v139-mkmlizer
Pipeline stage MKMLizer completed in 144.29s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.08s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-nemo-9330-v139
Waiting for inference service mistralai-mistral-nemo-9330-v139 to be ready
Inference service mistralai-mistral-nemo-9330-v139 ready after 120.49790096282959s
Pipeline stage MKMLDeployer completed in 120.92s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9128777980804443s
Received healthy response to inference request in 1.439941167831421s
Received healthy response to inference request in 1.7565195560455322s
Received healthy response to inference request in 0.6230835914611816s
Received healthy response to inference request in 1.3948190212249756s
5 requests
0 failed requests
5th percentile: 0.7774306774139405
10th percentile: 0.9317777633666993
20th percentile: 1.2404719352722169
30th percentile: 1.4038434505462647
40th percentile: 1.4218923091888427
50th percentile: 1.439941167831421
60th percentile: 1.5665725231170655
70th percentile: 1.69320387840271
80th percentile: 1.7877912044525146
90th percentile: 1.8503345012664796
95th percentile: 1.881606149673462
99th percentile: 1.906623468399048
mean time: 1.425448226928711
Pipeline stage StressChecker completed in 7.70s
Shutdown handler de-registered
mistralai-mistral-nemo_9330_v139 status is now deployed due to DeploymentManager action
mistralai-mistral-nemo_9330_v139 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Checking if service mistralai-mistral-nemo-9330-v136 is running
Shutdown handler not registered because Python interpreter is not running in the main thread
%s, retrying in %s seconds...
run pipeline %s
Checking if service mistralai-mistral-nemo-9330-v136 is running
run pipeline stage %s
%s, retrying in %s seconds...
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v136 is running
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7cfc342c-40c0-46fa-8f29-cf1f59e92032, 9d31ca6f-c076-4f3c-b666-73ea63fce053\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
%s, retrying in %s seconds...
Shutdown handler de-registered
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4f87369d-8d72-41d5-9f73-5595eee8409a, 073b8370-ce09-4dbd-9f35-2e64ce3322d2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3ccf0aed-a383-4cc1-bdcb-dee214b504ac, f17cd5ab-5ec2-4d17-a073-9e5bc2930dd5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:54 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'978487b6-8007-49f6-a4d1-a8abb08b2c3f, 194a79f1-ae1e-48df-a752-bf404974f6e0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'70653694-6910-4354-b422-e5ae59288a4b, 06187a9e-cac4-44dd-a223-5d58e17c1234\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
admin requested tearing down of mistralai-mistral-nemo_9330_v148
Checking if service mistralai-mistral-nemo-9330-v139 is running
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c4e61f18-e1b3-4dd1-ac35-f732c5110a35, abb1f4ac-3aa2-4bda-b3cb-bcb54f03d623\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:36:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
run pipeline stage %s
Shutdown handler de-registered
Running pipeline stage MKMLDeleter
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ce0f7a83-deea-4303-bbb2-27ed1498126e, cdb937af-4049-4ea7-84d8-4b0a4947cf7c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dd0f6718-47d2-4d60-9cde-50669d31b5a2, 5f6f2d57-0806-4f0f-8b8c-99977ea6ee26\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cd417667-5a9c-4efd-b647-6c40378213ba, 47a809d6-28ab-4d39-a2f2-49ef64163625\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:02 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6e5a9262-8cef-47cf-b74f-b23a06d11e68, 20859932-3045-475f-aae8-2ecef7ff7ca6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b103e19d-cbad-4b23-88a7-ca6b32712e20, cc857d23-983c-4265-92db-b33c547f3967\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1efeaddd-8cc8-4b0c-9954-3d6d7ee0cfba, b172f095-d0ab-4a80-8dfe-e874838342cf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6a3e5536-6e8a-4290-8861-2465b6e9990d, f13a3b90-4650-4668-9512-6e40e2009adf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ac6f9c0e-3ce1-4f6c-8dd6-3594f49c5078, e5feabb1-b43a-459e-86f8-2be734615d3f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'eda421f6-275d-451e-b914-8ab92ae7ef96, d6d8b713-cfa8-470e-a291-f8cf9807424f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'73a1a652-b04a-4bc6-a136-142136a4452f, 8c1eb117-75ad-41b4-ac8b-494af6baeed2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dece9eeb-9ae9-46c1-919f-58ac074795e3, 45372c14-6a44-4dd0-8944-d93a9ba544cb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7a9cefb0-13fe-44e6-9a55-116c5dad724a, a8c10c88-23c0-4051-853e-12e8070f4c16\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f2c5123b-f1a3-4432-bb9d-91639b8ae44a, 19bdfc76-4d43-457e-bea1-10b9b8a3d49a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c0be73d5-d4d1-4bc3-a2f3-ef76ecc38358, 48f29ee1-dbfc-4fe3-8e71-0ff32b1f8fe6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'143b9efd-0069-4187-b5ca-02e5db1fddd0, 5e772354-ba22-4e6e-a6b8-ad654a0565de\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8e83b3a7-8033-47f2-89f6-02cc105c5d46, 76686d1d-80f2-4b22-b031-ce7caccd18b5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:18 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'13b8b6e6-2696-4020-b39f-5fcfbccbdba6, c6402750-69be-4dbe-b4b8-56897139118e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9b76bdab-94d5-4925-98ac-566ec103a204, 96b0bed3-3b28-41a5-b228-b1940654e0e3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b580ed22-3c76-4909-887d-ad1309033c5b, e1124794-1dd3-4c38-9802-fd91d21b58a6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0c5009f5-4196-4928-a570-5dddc6509858, b31dad15-60a1-4f26-9830-88d014ef0e32\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'48d2d59a-8cf3-4ae4-a2fb-3cacfbd8232f, 73ee9687-2f17-4a3f-a665-817d8fc0e33f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of mistralai-mistral-nemo_9330_v139
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
%s, retrying in %s seconds...
Checking if service mistralai-mistral-nemo-9330-v139 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f72b8e02-8dd0-4812-999a-7f3ecad26b52, 18f3ddce-6d27-4577-a431-9fa0560fa87c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
mistralai-mistral-nemo_9330_v139 status is now torndown due to DeploymentManager action