nousresearch-meta-llama_4939

submission_id: nousresearch-meta-llama_4939_v69
developer_uid: chai_backend_admin
best_of: 8
celo_rating: 1213.4
display_name: nousresearch-meta-llama_4939_v69
family_friendly_score: 0.6080883428404271
family_friendly_standard_error: 0.007217590274931709
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: NousResearch/Meta-Llama-3.1-8B-Instruct
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: NousResearch/Meta-Llama-
model_name: nousresearch-meta-llama_4939_v69
model_num_parameters: 8030261248.0
model_repo: NousResearch/Meta-Llama-3.1-8B-Instruct
model_size: 8B
num_battles: 4302
num_wins: 1932
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-10-04T19:05:26+00:00
us_pacific_date: 2024-10-04
win_ratio: 0.4490934449093445
Download Preference Data
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nousresearch-meta-llama-4939-v69-mkmlizer
Waiting for job on nousresearch-meta-llama-4939-v69-mkmlizer to finish
nousresearch-meta-llama-4939-v69-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nousresearch-meta-llama-4939-v69-mkmlizer: ║     _____            __           __                                ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║       /___/                                                         ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  Version: 0.11.12                                                   ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  https://mk1.ai                                                     ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  The license key for the current software has been verified as      ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  belonging to:                                                      ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  Chai Research Corp.                                                ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
nousresearch-meta-llama-4939-v69-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v69-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nousresearch-meta-llama-4939-v69-mkmlizer: Downloaded to shared memory in 34.381s
nousresearch-meta-llama-4939-v69-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp891nemrl, device:0
nousresearch-meta-llama-4939-v69-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nousresearch-meta-llama-4939-v69-mkmlizer: quantized model in 29.587s
nousresearch-meta-llama-4939-v69-mkmlizer: Processed model NousResearch/Meta-Llama-3.1-8B-Instruct in 63.968s
nousresearch-meta-llama-4939-v69-mkmlizer: creating bucket guanaco-mkml-models
nousresearch-meta-llama-4939-v69-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nousresearch-meta-llama-4939-v69-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v69
nousresearch-meta-llama-4939-v69-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v69/config.json
nousresearch-meta-llama-4939-v69-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v69/special_tokens_map.json
nousresearch-meta-llama-4939-v69-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v69/tokenizer_config.json
nousresearch-meta-llama-4939-v69-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v69/tokenizer.json
nousresearch-meta-llama-4939-v69-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v69/flywheel_model.0.safetensors
nousresearch-meta-llama-4939-v69-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:08, 32.76it/s]
Loading 0:   4%|▍         | 13/291 [00:00<00:05, 53.59it/s]
Loading 0:   7%|▋         | 19/291 [00:00<00:05, 47.07it/s]
Loading 0:   8%|▊         | 24/291 [00:00<00:05, 46.31it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:04, 53.13it/s]
Loading 0:  13%|█▎        | 37/291 [00:00<00:05, 48.60it/s]
Loading 0:  15%|█▍        | 43/291 [00:00<00:04, 50.29it/s]
Loading 0:  17%|█▋        | 49/291 [00:00<00:04, 52.64it/s]
Loading 0:  19%|█▉        | 55/291 [00:01<00:04, 48.96it/s]
Loading 0:  21%|██        | 61/291 [00:01<00:04, 50.39it/s]
Loading 0:  23%|██▎       | 67/291 [00:01<00:04, 52.74it/s]
Loading 0:  25%|██▌       | 73/291 [00:01<00:04, 48.23it/s]
Loading 0:  27%|██▋       | 78/291 [00:01<00:04, 47.45it/s]
Loading 0:  29%|██▊       | 83/291 [00:01<00:06, 33.29it/s]
Loading 0:  30%|██▉       | 87/291 [00:01<00:06, 33.48it/s]
Loading 0:  32%|███▏      | 94/291 [00:02<00:04, 40.42it/s]
Loading 0:  34%|███▍      | 100/291 [00:02<00:04, 39.36it/s]
Loading 0:  36%|███▌      | 105/291 [00:02<00:04, 40.37it/s]
Loading 0:  38%|███▊      | 112/291 [00:02<00:03, 46.49it/s]
Loading 0:  41%|████      | 118/291 [00:02<00:03, 44.55it/s]
Loading 0:  42%|████▏     | 123/291 [00:02<00:03, 44.23it/s]
Loading 0:  45%|████▍     | 130/291 [00:02<00:03, 49.09it/s]
Loading 0:  47%|████▋     | 136/291 [00:03<00:03, 46.36it/s]
Loading 0:  48%|████▊     | 141/291 [00:03<00:03, 46.07it/s]
Loading 0:  51%|█████     | 148/291 [00:03<00:02, 50.74it/s]
Loading 0:  53%|█████▎    | 154/291 [00:03<00:02, 47.55it/s]
Loading 0:  55%|█████▍    | 159/291 [00:03<00:02, 47.18it/s]
Loading 0:  57%|█████▋    | 166/291 [00:03<00:02, 51.00it/s]
Loading 0:  59%|█████▉    | 172/291 [00:03<00:02, 47.45it/s]
Loading 0:  62%|██████▏   | 179/291 [00:03<00:02, 51.00it/s]
Loading 0:  64%|██████▎   | 185/291 [00:03<00:02, 51.55it/s]
Loading 0:  66%|██████▌   | 191/291 [00:04<00:02, 33.71it/s]
Loading 0:  67%|██████▋   | 196/291 [00:04<00:02, 35.62it/s]
Loading 0:  69%|██████▉   | 202/291 [00:04<00:02, 40.61it/s]
Loading 0:  71%|███████▏  | 208/291 [00:04<00:02, 40.60it/s]
Loading 0:  73%|███████▎  | 213/291 [00:04<00:01, 41.89it/s]
Loading 0:  76%|███████▌  | 220/291 [00:04<00:01, 47.67it/s]
Loading 0:  78%|███████▊  | 226/291 [00:05<00:01, 45.93it/s]
Loading 0:  79%|███████▉  | 231/291 [00:05<00:01, 45.83it/s]
Loading 0:  82%|████████▏ | 238/291 [00:05<00:01, 50.34it/s]
Loading 0:  84%|████████▍ | 244/291 [00:05<00:00, 47.71it/s]
Loading 0:  86%|████████▌ | 249/291 [00:05<00:00, 47.39it/s]
Loading 0:  88%|████████▊ | 256/291 [00:05<00:00, 51.55it/s]
Loading 0:  90%|█████████ | 262/291 [00:05<00:00, 48.34it/s]
Loading 0:  92%|█████████▏| 267/291 [00:05<00:00, 47.95it/s]
Loading 0:  94%|█████████▍| 274/291 [00:05<00:00, 52.81it/s]
Loading 0:  96%|█████████▌| 280/291 [00:06<00:00, 49.03it/s]
Loading 0:  98%|█████████▊| 286/291 [00:06<00:00, 42.93it/s]
Loading 0: 100%|██████████| 291/291 [00:11<00:00,  3.28it/s]
                                                            
Job nousresearch-meta-llama-4939-v69-mkmlizer completed after 83.61s with status: succeeded
Stopping job with name nousresearch-meta-llama-4939-v69-mkmlizer
Pipeline stage MKMLizer completed in 84.14s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service nousresearch-meta-llama-4939-v69
Waiting for inference service nousresearch-meta-llama-4939-v69 to be ready
Inference service nousresearch-meta-llama-4939-v69 ready after 110.10127115249634s
Pipeline stage MKMLDeployer completed in 110.63s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.3265113830566406s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.3195688724517822s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.5233452320098877s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.4640722274780273s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.6600866317749023s
5 requests
0 failed requests
5th percentile: 1.3484695434570313
10th percentile: 1.3773702144622804
20th percentile: 1.4351715564727783
30th percentile: 1.4759268283843994
40th percentile: 1.4996360301971436
50th percentile: 1.5233452320098877
60th percentile: 1.5780417919158936
70th percentile: 1.6327383518218994
80th percentile: 1.7933715820312501
90th percentile: 2.0599414825439455
95th percentile: 2.193226432800293
99th percentile: 2.299854393005371
mean time: 1.658716869354248
Pipeline stage StressChecker completed in 9.55s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
HTTP Request: %s %s "%s %d %s"
Pipeline stage TriggerMKMLProfilingPipeline completed in 358.24s
Shutdown handler de-registered
Shutting down server chaiverse_console.
nousresearch-meta-llama_4939_v69 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1112ead9-2f8f-40e8-88bd-023c3a8f3c1c, f39d4a66-00f4-419a-847b-e5402eef8702\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f91b75d1-b08f-4134-b20b-292ea379b55e, 072c1a5e-8f47-41fe-a72c-4ed1eb0ce291\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
admin requested tearing down of nvidia-mistral-nemo-mini_3107_v1
Checking if service nousresearch-meta-llama-4939-v69 is running
Shutdown handler not registered because Python interpreter is not running in the main thread
%s, retrying in %s seconds...
run pipeline %s
Checking if service nousresearch-meta-llama-4939-v69 is running
run pipeline stage %s
%s, retrying in %s seconds...
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c35db8f0-ff4e-445d-a9ab-108f9998ffaf, 95ae5b43-0ab6-415a-91e4-d49b32f533fa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
%s, retrying in %s seconds...
Shutdown handler de-registered
Checking if service nvidia-mistral-nemo-mini-3107-v1 is running
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b7889422-2858-488e-93c3-c8ff9709f673, 6299ff7f-a728-443a-9511-d54b7b96020f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'61486207-111a-4326-87c1-785f00fa455c, c76ac865-a481-47ca-8937-abfcfc162c9d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:36:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'95cae9de-1d69-45be-8b9d-cfe68fa0d0f2, 450efefd-520a-4f13-8c8f-90ae554171c7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'563be451-07e1-4d91-87d5-c210d289b927, 8204d649-043e-4be8-8c65-680d82f2de09\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7039a59b-c6d0-4b40-ba20-0621a4287694, 4e3150c5-8d94-4cbb-87bc-f9912fafdaf0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4090376b-d5e6-4a43-90e1-5fdd305decfe, 463dc780-8427-450b-a6c4-ffa44cb5e1c1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1d4bd254-bc02-4187-8a80-f36317edbd40, 38b12576-e491-428a-9201-389ccb474a43\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:22 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'35e8dfd5-e996-4292-ab08-30f587fea8e0, 542cbc39-9c76-4b46-b215-3281f35fdcac\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b9f1eab3-ebbe-4d10-a889-6eb1b981ea00, 61c0bd50-e868-458a-ac86-f76424b7c751\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'711373ea-9785-4b0f-9fde-ec69bd5685b5, 1d5fd28a-a628-4fce-b6f5-f1ad3205acf6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'846e97f9-aa2b-43eb-af7f-acc0c0823e41, 60a5bf8f-ed25-4571-bf3a-d6c3bed0f3a9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6a8ad234-d757-4c92-adea-6fb630c41110, c739f80f-d5cb-4c88-b1e8-51b9acb07ff4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e281a231-d5df-425f-8f94-af153a90a8a1, 4feb6503-8eb4-4fa0-b3c6-f96004315785\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'993ac253-cdfb-498d-a8e5-15d3306fb6d7, acfd8c01-f293-4f2d-8a81-6c8eb344198a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:18 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'eee07295-5ba4-4006-8992-00fe1099ca49, 7292d27f-e10a-4c81-8162-a37bdcb61e57\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2be38354-0ff7-4719-bccf-47246269d086, b444da47-892c-4624-9213-b3df75cc2b8e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:25 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'15b15a08-a821-4cbe-bf80-8714f12d487c, 5bdea2eb-4a92-4a3a-89fa-a25b46c375ff\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:25 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a3c5c0af-b759-4f31-a81c-6a53da48e574, 60c94b80-5b05-411b-a392-68d98bd6ae9d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4516e5d6-ea46-4989-9c79-b35884c15f65, 59408273-2886-49fc-8dd7-22310f10bb2e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'78db5b39-06e9-4eea-9a7e-06c3c4fe7feb, 15d07412-4ccb-4451-aba1-23392f47144d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b7d39af0-858d-455d-b8fe-dc90187041e5, f16d6002-8707-4f6e-aaf1-0c4bd095dfbf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3b5177d6-bbbb-4d0e-8da7-ac26a236ccd8, bf8f5b0b-3579-42ec-aaac-5fab50fb5956\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b6af32bb-c3d9-4100-bdbe-734acccd1f2c, 34c57a3a-bd1d-440b-a86b-355a18efb637\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nousresearch-meta-llama_4939_v69
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
%s, retrying in %s seconds...
Checking if service nousresearch-meta-llama-4939-v69 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'60a7df50-469a-4ca4-b1a8-65659e8da265, 30de0ded-b233-4b28-b19c-b666833936cd\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
nousresearch-meta-llama_4939_v69 status is now torndown due to DeploymentManager action