nousresearch-meta-llama_4939

submission_id: nousresearch-meta-llama_4939_v70
developer_uid: chai_backend_admin
best_of: 8
celo_rating: 1223.06
display_name: nousresearch-meta-llama_4939_v70
family_friendly_score: 0.610581188997338
family_friendly_standard_error: 0.007247228773402892
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: NousResearch/Meta-Llama-3.1-8B-Instruct
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: NousResearch/Meta-Llama-
model_name: nousresearch-meta-llama_4939_v70
model_num_parameters: 8030261248.0
model_repo: NousResearch/Meta-Llama-3.1-8B-Instruct
model_size: 8B
num_battles: 4241
num_wins: 1973
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-10-04T19:15:47+00:00
us_pacific_date: 2024-10-04
win_ratio: 0.465220466871021
Download Preference Data
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nousresearch-meta-llama-4939-v70-mkmlizer
Waiting for job on nousresearch-meta-llama-4939-v70-mkmlizer to finish
nousresearch-meta-llama-4939-v70-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nousresearch-meta-llama-4939-v70-mkmlizer: ║     _____            __           __                                ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║       /___/                                                         ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  Version: 0.11.12                                                   ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  https://mk1.ai                                                     ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  The license key for the current software has been verified as      ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  belonging to:                                                      ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  Chai Research Corp.                                                ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
nousresearch-meta-llama-4939-v70-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v70-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nousresearch-meta-llama-4939-v70-mkmlizer: Downloaded to shared memory in 34.604s
nousresearch-meta-llama-4939-v70-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp69fl8lxv, device:0
nousresearch-meta-llama-4939-v70-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nousresearch-meta-llama-4939-v70-mkmlizer: quantized model in 29.533s
nousresearch-meta-llama-4939-v70-mkmlizer: Processed model NousResearch/Meta-Llama-3.1-8B-Instruct in 64.137s
nousresearch-meta-llama-4939-v70-mkmlizer: creating bucket guanaco-mkml-models
nousresearch-meta-llama-4939-v70-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nousresearch-meta-llama-4939-v70-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v70
nousresearch-meta-llama-4939-v70-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v70/config.json
nousresearch-meta-llama-4939-v70-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v70/special_tokens_map.json
nousresearch-meta-llama-4939-v70-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v70/tokenizer_config.json
nousresearch-meta-llama-4939-v70-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v70/flywheel_model.0.safetensors
nousresearch-meta-llama-4939-v70-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:08, 33.87it/s]
Loading 0:   4%|▍         | 13/291 [00:00<00:05, 55.58it/s]
Loading 0:   7%|▋         | 19/291 [00:00<00:05, 49.48it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:05, 51.75it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:04, 54.32it/s]
Loading 0:  13%|█▎        | 37/291 [00:00<00:05, 49.98it/s]
Loading 0:  15%|█▍        | 43/291 [00:00<00:04, 51.78it/s]
Loading 0:  17%|█▋        | 50/291 [00:01<00:05, 47.55it/s]
Loading 0:  20%|█▉        | 58/291 [00:01<00:04, 55.54it/s]
Loading 0:  22%|██▏       | 64/291 [00:01<00:04, 50.82it/s]
Loading 0:  24%|██▍       | 70/291 [00:01<00:04, 51.77it/s]
Loading 0:  26%|██▌       | 76/291 [00:01<00:04, 53.70it/s]
Loading 0:  28%|██▊       | 82/291 [00:01<00:04, 49.52it/s]
Loading 0:  30%|███       | 88/291 [00:01<00:05, 34.77it/s]
Loading 0:  32%|███▏      | 94/291 [00:02<00:05, 39.19it/s]
Loading 0:  34%|███▍      | 100/291 [00:02<00:04, 38.78it/s]
Loading 0:  36%|███▌      | 105/291 [00:02<00:04, 39.86it/s]
Loading 0:  38%|███▊      | 112/291 [00:02<00:03, 45.89it/s]
Loading 0:  41%|████      | 118/291 [00:02<00:03, 44.00it/s]
Loading 0:  42%|████▏     | 123/291 [00:02<00:03, 44.04it/s]
Loading 0:  45%|████▍     | 130/291 [00:02<00:03, 49.33it/s]
Loading 0:  47%|████▋     | 136/291 [00:02<00:03, 46.50it/s]
Loading 0:  48%|████▊     | 141/291 [00:03<00:03, 46.20it/s]
Loading 0:  51%|█████     | 148/291 [00:03<00:02, 51.45it/s]
Loading 0:  53%|█████▎    | 154/291 [00:03<00:02, 47.72it/s]
Loading 0:  55%|█████▍    | 159/291 [00:03<00:02, 46.91it/s]
Loading 0:  57%|█████▋    | 166/291 [00:03<00:02, 51.84it/s]
Loading 0:  59%|█████▉    | 172/291 [00:03<00:02, 48.82it/s]
Loading 0:  62%|██████▏   | 179/291 [00:03<00:02, 52.25it/s]
Loading 0:  64%|██████▎   | 185/291 [00:03<00:01, 53.73it/s]
Loading 0:  66%|██████▌   | 191/291 [00:04<00:02, 34.85it/s]
Loading 0:  67%|██████▋   | 196/291 [00:04<00:02, 36.61it/s]
Loading 0:  69%|██████▉   | 202/291 [00:04<00:02, 40.77it/s]
Loading 0:  71%|███████▏  | 208/291 [00:04<00:02, 40.41it/s]
Loading 0:  73%|███████▎  | 213/291 [00:04<00:01, 41.52it/s]
Loading 0:  76%|███████▌  | 220/291 [00:04<00:01, 47.37it/s]
Loading 0:  78%|███████▊  | 226/291 [00:04<00:01, 44.92it/s]
Loading 0:  79%|███████▉  | 231/291 [00:05<00:01, 44.65it/s]
Loading 0:  82%|████████▏ | 238/291 [00:05<00:01, 49.90it/s]
Loading 0:  84%|████████▍ | 244/291 [00:05<00:00, 47.64it/s]
Loading 0:  86%|████████▌ | 249/291 [00:05<00:00, 47.09it/s]
Loading 0:  88%|████████▊ | 256/291 [00:05<00:00, 51.67it/s]
Loading 0:  90%|█████████ | 262/291 [00:05<00:00, 48.49it/s]
Loading 0:  92%|█████████▏| 267/291 [00:05<00:00, 47.61it/s]
Loading 0:  94%|█████████▍| 274/291 [00:05<00:00, 51.63it/s]
Loading 0:  96%|█████████▌| 280/291 [00:06<00:00, 48.09it/s]
Loading 0:  98%|█████████▊| 285/291 [00:06<00:00, 47.55it/s]
Loading 0: 100%|█████████▉| 290/291 [00:11<00:00,  3.12it/s]
                                                            
Job nousresearch-meta-llama-4939-v70-mkmlizer completed after 83.73s with status: succeeded
Stopping job with name nousresearch-meta-llama-4939-v70-mkmlizer
Pipeline stage MKMLizer completed in 84.32s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.20s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service nousresearch-meta-llama-4939-v70
Waiting for inference service nousresearch-meta-llama-4939-v70 to be ready
Inference service nousresearch-meta-llama-4939-v70 ready after 122.41333675384521s
Pipeline stage MKMLDeployer completed in 122.98s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.9250094890594482s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.2422709465026855s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.5825018882751465s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.5606958866119385s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.3055472373962402s
5 requests
0 failed requests
5th percentile: 1.2549262046813965
10th percentile: 1.2675814628601074
20th percentile: 1.2928919792175293
30th percentile: 1.35657696723938
40th percentile: 1.458636426925659
50th percentile: 1.5606958866119385
60th percentile: 1.5694182872772218
70th percentile: 1.5781406879425048
80th percentile: 1.6510034084320069
90th percentile: 1.7880064487457275
95th percentile: 1.8565079689025878
99th percentile: 1.9113091850280761
mean time: 1.5232050895690918
Pipeline stage StressChecker completed in 9.09s
Shutdown handler de-registered
nousresearch-meta-llama_4939_v70 status is now deployed due to DeploymentManager action
nousresearch-meta-llama_4939_v70 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of nousresearch-meta-llama_4939_v70
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4c70533e-52c9-424a-85b0-f6be1a876109, 1de7a7e4-b754-4da2-829f-f1ed07a39a74\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler not registered because Python interpreter is not running in the main thread
Shutdown handler de-registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b0c93bc1-a3f0-418e-bcf5-d6ef69f67358, bfc7a492-695c-49cb-a926-0fc2f19353f1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2f4cfef8-e90a-4766-bcce-e845c3f43ad9, da76faea-d659-450d-bdf1-f0e3c0dc98b2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0e2185da-a302-4531-9468-8b323b22d780, bb77af9c-d87c-49b9-b310-bab9519bdf81\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:02 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a4def384-dcba-4a28-a88f-8776a3a444a2, f6ca8bee-9de8-460f-a056-573e4130c91a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'64c681f8-ace7-4e3a-8618-b2b9543ce175, 2f72fc9f-95eb-4960-b6ab-064f2110299e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:36:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e5833b89-08a5-4366-87bb-f01ce91f63cc, 8cb248bb-b3be-41f7-84cc-8424ff5d5fdb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d5ee5657-0785-43d2-922f-4f04e64efa13, 028bcf86-d830-4161-ae0f-e5615dcb5a5d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'827c1327-4053-43b8-a333-4d05bfe55293, 487c91c2-680b-46ca-bc69-ccb95ad2f366\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'082e7771-adc0-4a1c-8e09-7c7b246e3256, a89fc0ad-a63b-4ecb-9ccf-9768ada6583c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'59f375ca-9dc3-4050-9218-79f73d14cdf8, ef73a66c-3bc5-407d-8c92-51b3706329fa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:23 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b74352b3-d024-43bf-976a-85a95a5b6441, 0c2731af-c180-4ea1-936a-cefd6ff506bc\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'df75fe83-f99f-4f98-bf2f-48e92e5ed125, a3ed92ef-62b9-4392-aaaf-cacaa0bc3b5c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'15996e12-44d7-40db-9d0f-13a62dfa4d87, 7156c19c-2ac7-45de-8f44-81aba89c22d1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6d025de1-3aab-40db-86b9-2c56754f4f68, 92de5e3d-178b-4ba6-b49f-6ccc0e6a689e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7b381257-ab62-4b27-8a00-c57ea0f0580d, 7fa326de-d9ac-4774-9a00-118f3a106bd1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'32391912-65b1-4392-af91-191113c2249b, 63a72e77-d516-4075-9c11-5df8513d9f0f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2a158115-dea1-474d-bf79-ddd5dddc38e1, 2e8470ca-d720-4d62-b5b3-0719ed0c0a0b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'84d28dc5-a6e0-4e73-86ef-34e9d3c9214d, f5481b06-00e8-45d1-a777-287d178fa244\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bf374d45-2ae3-4546-9502-0087d58b64f8, 1b0f3d4e-45a6-43e7-abdb-4194f6063c5c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b0208526-828a-4fb7-b88c-ff131fb77f7d, fc59f4b9-1f13-44ef-9001-176bd41813de\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'716d7764-598e-46b3-9b6c-07528d65637f, 4745fc68-a981-4f16-876e-6bef7046b839\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'681e3ed8-334e-4497-8e91-8c2295fe1f65, 5f3653dc-0d6b-4f6e-ad41-da1b7f0971a2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:41 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6cd03cbd-d59d-4a73-bb02-61444570d1e7, 785b88ce-91e8-4dca-8879-fa9ce3c2f32b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'59817cfd-6f54-4dd8-9c46-6a92bc5fa1eb, 9e6789df-8e36-4fa1-87f5-ff9273888d8b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'476b472c-db30-4e0f-87e9-1842806bbf7b, e2189078-71f1-4dae-843c-0de6afc6a5e5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'455d8d47-2e88-47ff-94b4-30e2349182c0, 3eb5f36a-fa4d-4d61-9261-c2a50643216a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v70
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v70 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7c9b6dc0-31e3-4e9c-ab13-a01ea8b6c7d1, ac21de7b-f050-4d99-a3b8-0131d8414757\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
nousresearch-meta-llama_4939_v70 status is now torndown due to DeploymentManager action