developer_uid: chace9580
submission_id: jic062-nemo-v1-6_v4
model_name: jic062-nemo-v1-6_v4
model_group: jic062/Nemo-v1.6
status: torndown
timestamp: 2024-09-29T01:24:56+00:00
num_battles: 7967
num_wins: 4139
celo_rating: 1266.22
family_friendly_score: 0.5972056285676357
family_friendly_standard_error: 0.005617381448388459
submission_type: basic
model_repo: jic062/Nemo-v1.6
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: jic062-nemo-v1-6_v4
is_internal_developer: False
language_model: jic062/Nemo-v1.6
model_size: 13B
ranking_group: single
us_pacific_date: 2024-09-28
win_ratio: 0.5195180117986695
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '/s', '[/INST]'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '[INST]system\n{memory}[/INST]\n', 'prompt_template': '[INST]user\n{prompt}[/INST]\n', 'bot_template': '[INST]assistant\n{bot_name}: {message}[/INST]\n', 'user_template': '[INST]user\n{user_name}: {message}[/INST]\n', 'response_template': '[INST]assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-nemo-v1-6-v4-mkmlizer
Waiting for job on jic062-nemo-v1-6-v4-mkmlizer to finish
jic062-nemo-v1-6-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-nemo-v1-6-v4-mkmlizer: ║     _____            __           __                                ║
jic062-nemo-v1-6-v4-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
jic062-nemo-v1-6-v4-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
jic062-nemo-v1-6-v4-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
jic062-nemo-v1-6-v4-mkmlizer: ║       /___/                                                         ║
jic062-nemo-v1-6-v4-mkmlizer: ║                                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Version: 0.11.12                                                   ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
jic062-nemo-v1-6-v4-mkmlizer: ║  https://mk1.ai                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ║                                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ║  The license key for the current software has been verified as      ║
jic062-nemo-v1-6-v4-mkmlizer: ║  belonging to:                                                      ║
jic062-nemo-v1-6-v4-mkmlizer: ║                                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Chai Research Corp.                                                ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
jic062-nemo-v1-6-v4-mkmlizer: ║                                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Inference service jic062-nemo-v1-8-v1 ready after 230.64079999923706s
Pipeline stage MKMLDeployer completed in 231.16s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3941752910614014s
Received healthy response to inference request in 1.8138670921325684s
Inference service jic062-nemo-v1-8-v2 ready after 230.6058521270752s
Received healthy response to inference request in 2.036547899246216s
Pipeline stage MKMLDeployer completed in 231.05s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.058387279510498s
Received healthy response to inference request in 2.402942180633545s
Received healthy response to inference request in 1.9270868301391602s
5 requests
0 failed requests
5th percentile: 1.8365110397338866
10th percentile: 1.8591549873352051
Received healthy response to inference request in 1.7296473979949951s
20th percentile: 1.9044428825378419
30th percentile: 1.9489790439605712
40th percentile: 1.9927634716033935
50th percentile: 2.036547899246216
60th percentile: 2.045283651351929
70th percentile: 2.054019403457642
80th percentile: 2.125544881820679
90th percentile: 2.25986008644104
95th percentile: 2.3270176887512206
99th percentile: 2.380743770599365
mean time: 2.0460128784179688
Pipeline stage StressChecker completed in 12.39s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
jic062-nemo-v1-6-v4-mkmlizer: Downloaded to shared memory in 62.361s
jic062-nemo-v1-6-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpuqdz21op, device:0
jic062-nemo-v1-6-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Received healthy response to inference request in 1.7409923076629639s
Received healthy response to inference request in 1.69749116897583s
Received healthy response to inference request in 1.7477772235870361s
5 requests
0 failed requests
5th percentile: 1.703922414779663
10th percentile: 1.7103536605834961
20th percentile: 1.7232161521911622
30th percentile: 1.7319163799285888
40th percentile: 1.7364543437957765
50th percentile: 1.7409923076629639
60th percentile: 1.7437062740325928
70th percentile: 1.7464202404022218
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.44s
80th percentile: 1.878810214996338
Shutdown handler de-registered
jic062-nemo-v1-8_v1 status is now deployed due to DeploymentManager action
95th percentile: 2.271909189224243
99th percentile: 2.3767355823516847
mean time: 1.863770055770874
Pipeline stage StressChecker completed in 11.74s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.97s
Shutdown handler de-registered
jic062-nemo-v1-8_v2 status is now deployed due to DeploymentManager action
jic062-nemo-v1-6-v4-mkmlizer: quantized model in 36.818s
jic062-nemo-v1-6-v4-mkmlizer: Processed model jic062/Nemo-v1.6 in 99.179s
jic062-nemo-v1-6-v4-mkmlizer: creating bucket guanaco-mkml-models
jic062-nemo-v1-6-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-nemo-v1-6-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-nemo-v1-6-v4
jic062-nemo-v1-6-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-nemo-v1-6-v4/config.json
jic062-nemo-v1-6-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-nemo-v1-6-v4/special_tokens_map.json
jic062-nemo-v1-6-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-nemo-v1-6-v4/tokenizer_config.json
jic062-nemo-v1-6-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-nemo-v1-6-v4/tokenizer.json
jic062-nemo-v1-6-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/jic062-nemo-v1-6-v4/flywheel_model.0.safetensors
jic062-nemo-v1-6-v4-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 29.08it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 47.08it/s]
Loading 0:   5%|▍         | 18/363 [00:00<00:07, 45.52it/s]
Loading 0:   6%|▋         | 23/363 [00:00<00:09, 35.99it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:07, 46.23it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 45.58it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 44.66it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 49.74it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 44.71it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:09, 31.78it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:09, 31.19it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 37.58it/s]
Loading 0:  21%|██        | 77/363 [00:01<00:07, 40.11it/s]
Loading 0:  23%|██▎       | 82/363 [00:02<00:07, 36.48it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:06, 43.67it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 43.51it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:05, 44.44it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:06, 42.70it/s]
Loading 0:  30%|███       | 110/363 [00:02<00:05, 42.68it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 41.86it/s]
Loading 0:  33%|███▎      | 120/363 [00:02<00:05, 40.52it/s]
Loading 0:  35%|███▍      | 126/363 [00:03<00:05, 43.51it/s]
Loading 0:  36%|███▌      | 131/363 [00:03<00:05, 45.09it/s]
Loading 0:  37%|███▋      | 136/363 [00:03<00:06, 37.54it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:07, 30.99it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 31.60it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:07, 30.40it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 36.03it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:05, 35.57it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:04, 39.47it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:04, 39.80it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 39.59it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 43.44it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:04, 42.89it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:04, 41.96it/s]
Loading 0:  55%|█████▍    | 198/363 [00:04<00:03, 43.57it/s]
Loading 0:  56%|█████▌    | 203/363 [00:05<00:04, 36.55it/s]
Loading 0:  58%|█████▊    | 210/363 [00:05<00:03, 43.14it/s]
Loading 0:  59%|█████▉    | 215/363 [00:05<00:03, 40.63it/s]
Loading 0:  61%|██████    | 220/363 [00:05<00:03, 41.60it/s]
Loading 0:  62%|██████▏   | 225/363 [00:05<00:05, 26.57it/s]
Loading 0:  63%|██████▎   | 230/363 [00:06<00:04, 29.16it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 36.52it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 37.52it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:02, 38.87it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 41.26it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:03, 34.28it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 41.39it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 41.86it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 43.17it/s]
Loading 0:  77%|███████▋  | 280/363 [00:07<00:01, 42.21it/s]
Loading 0:  79%|███████▊  | 285/363 [00:07<00:01, 41.43it/s]
Loading 0:  80%|████████  | 292/363 [00:07<00:01, 46.29it/s]
Loading 0:  82%|████████▏ | 298/363 [00:07<00:01, 45.03it/s]
Loading 0:  84%|████████▎ | 304/363 [00:14<00:21,  2.73it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:15,  3.44it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:11,  4.40it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:06,  7.14it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:03,  9.57it/s]
Loading 0:  91%|█████████ | 331/363 [00:15<00:02, 12.07it/s]
Loading 0:  93%|█████████▎| 338/363 [00:15<00:01, 16.83it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:00, 20.72it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 24.03it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 30.66it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 32.96it/s]
                                                            

Job jic062-nemo-v1-6-v4-mkmlizer completed after 124.74s with status: succeeded
Stopping job with name jic062-nemo-v1-6-v4-mkmlizer
Pipeline stage MKMLizer completed in 125.23s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.08s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service jic062-nemo-v1-6-v4
Waiting for inference service jic062-nemo-v1-6-v4 to be ready
Inference service jic062-nemo-v1-6-v4 ready after 220.5620551109314s
Pipeline stage MKMLDeployer completed in 220.89s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1908164024353027s
Received healthy response to inference request in 1.7329609394073486s
Received healthy response to inference request in 1.7361228466033936s
Received healthy response to inference request in 1.695289134979248s
Received healthy response to inference request in 1.711799144744873s
5 requests
0 failed requests
5th percentile: 1.698591136932373
10th percentile: 1.701893138885498
20th percentile: 1.7084971427917481
30th percentile: 1.7160315036773681
40th percentile: 1.7244962215423585
50th percentile: 1.7329609394073486
60th percentile: 1.7342257022857666
70th percentile: 1.7354904651641845
80th percentile: 1.8270615577697755
90th percentile: 2.0089389801025392
95th percentile: 2.099877691268921
99th percentile: 2.1726286602020264
mean time: 1.8133976936340332
Pipeline stage StressChecker completed in 9.84s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.06s
Shutdown handler de-registered
jic062-nemo-v1-6_v4 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service jic062-nemo-v1-6-v4-profiler
Waiting for inference service jic062-nemo-v1-6-v4-profiler to be ready
Tearing down inference service jic062-nemo-v1-6-v4-profiler
%s, retrying in %s seconds...
Creating inference service jic062-nemo-v1-6-v4-profiler
Waiting for inference service jic062-nemo-v1-6-v4-profiler to be ready
Tearing down inference service jic062-nemo-v1-6-v4-profiler
%s, retrying in %s seconds...
Creating inference service jic062-nemo-v1-6-v4-profiler
Waiting for inference service jic062-nemo-v1-6-v4-profiler to be ready
Tearing down inference service jic062-nemo-v1-6-v4-profiler
clean up pipeline due to error=%s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.16s
Shutdown handler de-registered
jic062-nemo-v1-6_v4 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f60e13cc-5dd4-43f7-9e0e-2a8a466d8cc7, 84ccaf87-0432-46a3-8067-4d0599ac3c75\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'124be593-62c9-4781-940b-c01060abd2e9, 1e4343cf-07ef-446d-8f8c-0f1e51247f5d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'09658a9f-e50f-44c7-a4bd-28da32dcabbd, 1c5237cc-f024-47f7-ae82-644409d56166\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:52 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'110f972d-b05d-4c1e-bba2-fd5aed220c0c, 4339144a-8a6d-47d0-b0a2-4f4b9c9a56a7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6d64ad6b-bfe8-4bbc-8d1b-eac418503896, 4aa7eede-7d41-4a0f-87e1-00cbb82ce6a5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:35:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'68a0743e-48bc-4f25-a75f-423f26e7b522, 206b05ea-f056-4c5c-a625-202ffd345afe\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:40:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2f63bd55-4ea6-4e90-b7b0-b675c1733f0d, 246d3b53-4c5d-4a40-b725-13c86f2660b6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:45:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c006dfe3-c816-44c4-8fd2-9790d658430f, c1a5b2b9-7d43-4299-99d8-19b67ba4578a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:50:57 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e35cb9e9-6a77-4dd6-aaf0-5cf679197069, 34cc3b90-4b63-4144-a2db-4fede469075f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:55:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'62f81214-782f-42c7-af02-1f8b8d516d97, 577d0955-9dc6-4865-a321-4d6f5cdbafe5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4f3edca4-844d-4019-8f6d-57fb9ed29038, 4c15d035-b536-4032-9ce3-61644167da9a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:02 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e3ba489d-b990-45b3-8017-01a8e9553ead, 473f4886-0f3d-4502-ad68-461d9558bc4d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'db7945bb-fa98-416b-a2d2-de7fa9a6bbad, b63d9432-ae7d-4282-b80e-e72619ca9413\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a5127169-f3d9-4add-8c39-84713cdb1cdc, dfb986a9-912a-42f9-be87-fe995615123a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'18b5acd2-2e93-4f85-bebc-55136f4d59db, ae0b64f0-6be4-403b-a109-a644547e9b0c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d7c4df6e-6f71-4e27-b6e3-fb921892e1ff, 6343b965-31cc-4519-82eb-c117acbf08fa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:07 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'475e8062-9224-484a-a976-3447cf45a164, 7439f9d0-c237-4746-abac-12ebaf37f570\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0674d309-012f-4f9e-9ac6-8fc3829fc80a, 3933ba63-87b8-43fa-bd79-c41233636ad1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9c2d684f-ff97-4e3f-8711-3eaba7fbe0c6, 5af182db-bc96-4658-a89b-7abaeadce39c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'91dd02a8-846d-4ece-bf32-aae17ef34ba6, 64e6c095-9a6a-4191-9c33-11f1bd9a4459\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b4a1d338-9440-41eb-af79-f554120fea13, 71ce69fc-a293-4d84-8291-66ea7ec84e3b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'486b5d3a-2220-49af-93ef-7b345111b335, 3539f4b8-5d15-49ce-916a-db9affe67b10\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'88a4ddfb-cc47-4ea6-9761-6da94c1caa19, 32ba33cd-0e92-49d1-8325-3e7eec3aa25f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c559ce3f-2664-4887-acaf-2d758bc8f762, f8319601-517b-4d8d-8085-9085b3512fcb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3f8b0bf6-c504-4b45-9383-3a90b258ff0a, 0d6fd248-c2ff-44ba-89ce-7ae0d516947b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3ddae60f-866b-4f24-acf0-0cbf3ba65c8b, 80b4e810-21f4-47d1-a60f-d1eaa42e80ae\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'aa4995fd-8d84-4819-8d3a-2c06d9f1a2a4, 4cc6c1bb-a18f-424c-95b1-d67ecaeb3d88\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
jic062-nemo-v1-6_v4 status is now torndown due to DeploymentManager action