developer_uid: rinen0721
submission_id: rinen0721-llama0929_v2
model_name: rinen0721-llama0929_v1
model_group: rinen0721/llama0929
status: torndown
timestamp: 2024-09-30T03:39:09+00:00
num_battles: 3969
num_wins: 1812
celo_rating: 1219.99
family_friendly_score: 0.5885850694444444
family_friendly_standard_error: 0.007919890082081115
submission_type: basic
model_repo: rinen0721/llama0929
model_architecture: LlamaForCausalLM
model_num_parameters: 8030261248.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: rinen0721-llama0929_v1
ineligible_reason: num_battles<5000
is_internal_developer: False
language_model: rinen0721/llama0929
model_size: 8B
ranking_group: single
us_pacific_date: 2024-09-29
win_ratio: 0.4565381708238851
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rinen0721-llama0929-v2-mkmlizer
Waiting for job on rinen0721-llama0929-v2-mkmlizer to finish
rinen0721-llama0929-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rinen0721-llama0929-v2-mkmlizer: ║     _____            __           __                                ║
rinen0721-llama0929-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rinen0721-llama0929-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rinen0721-llama0929-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rinen0721-llama0929-v2-mkmlizer: ║       /___/                                                         ║
rinen0721-llama0929-v2-mkmlizer: ║                                                                     ║
rinen0721-llama0929-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
rinen0721-llama0929-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rinen0721-llama0929-v2-mkmlizer: ║  https://mk1.ai                                                     ║
rinen0721-llama0929-v2-mkmlizer: ║                                                                     ║
rinen0721-llama0929-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
rinen0721-llama0929-v2-mkmlizer: ║  belonging to:                                                      ║
rinen0721-llama0929-v2-mkmlizer: ║                                                                     ║
rinen0721-llama0929-v2-mkmlizer: ║  Chai Research Corp.                                                ║
rinen0721-llama0929-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rinen0721-llama0929-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rinen0721-llama0929-v2-mkmlizer: ║                                                                     ║
rinen0721-llama0929-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rinen0721-llama0924-v2-mkmlizer
Waiting for job on rinen0721-llama0924-v2-mkmlizer to finish
rinen0721-llama0929-v2-mkmlizer: Downloaded to shared memory in 23.780s
rinen0721-llama0929-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpdgzd4_4v, device:0
rinen0721-llama0929-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rinen0721-llama0929-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py:55: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
rinen0721-llama0929-v2-mkmlizer:   tensors = torch.load(model_shard_filename, map_location=torch.device(self.device), mmap=True)
rinen0721-llama0929-v2-mkmlizer: quantized model in 25.658s
rinen0721-llama0929-v2-mkmlizer: Processed model rinen0721/llama0929 in 49.438s
rinen0721-llama0929-v2-mkmlizer: creating bucket guanaco-mkml-models
rinen0721-llama0929-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rinen0721-llama0929-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rinen0721-llama0929-v2
rinen0721-llama0929-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rinen0721-llama0929-v2/config.json
rinen0721-llama0929-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rinen0721-llama0929-v2/special_tokens_map.json
rinen0721-llama0929-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rinen0721-llama0929-v2/tokenizer_config.json
rinen0721-llama0929-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rinen0721-llama0929-v2/tokenizer.json
rinen0721-llama0929-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rinen0721-llama0929-v2/flywheel_model.0.safetensors
rinen0721-llama0929-v2-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 7/291 [00:00<00:05, 48.93it/s]
Loading 0:   6%|▌         | 17/291 [00:00<00:03, 73.90it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:03, 74.90it/s]
Loading 0:  12%|█▏        | 34/291 [00:00<00:03, 76.79it/s]
Loading 0:  15%|█▍        | 43/291 [00:00<00:03, 79.56it/s]
Loading 0:  18%|█▊        | 52/291 [00:00<00:02, 82.11it/s]
Loading 0:  21%|██        | 61/291 [00:00<00:02, 83.80it/s]
Loading 0:  24%|██▍       | 70/291 [00:00<00:02, 84.39it/s]
Loading 0:  27%|██▋       | 79/291 [00:00<00:02, 85.48it/s]
Loading 0:  30%|███       | 88/291 [00:02<00:09, 21.93it/s]
Loading 0:  33%|███▎      | 97/291 [00:02<00:06, 28.31it/s]
Loading 0:  36%|███▋      | 106/291 [00:02<00:05, 35.73it/s]
Loading 0:  40%|███▉      | 116/291 [00:02<00:03, 45.14it/s]
Loading 0:  45%|████▍     | 130/291 [00:02<00:02, 56.85it/s]
Loading 0:  48%|████▊     | 139/291 [00:02<00:02, 62.33it/s]
Loading 0:  51%|█████     | 148/291 [00:02<00:02, 66.82it/s]
Loading 0:  54%|█████▍    | 157/291 [00:02<00:01, 70.71it/s]
Loading 0:  57%|█████▋    | 166/291 [00:02<00:01, 74.73it/s]
Loading 0:  60%|██████    | 175/291 [00:03<00:01, 77.27it/s]
Loading 0:  64%|██████▍   | 186/291 [00:03<00:01, 85.44it/s]
Loading 0:  67%|██████▋   | 196/291 [00:04<00:03, 23.81it/s]
Loading 0:  70%|███████   | 205/291 [00:04<00:02, 29.83it/s]
Loading 0:  74%|███████▎  | 214/291 [00:04<00:02, 36.81it/s]
Loading 0:  77%|███████▋  | 223/291 [00:04<00:01, 43.95it/s]
Loading 0:  80%|███████▉  | 232/291 [00:04<00:01, 50.67it/s]
Loading 0:  83%|████████▎ | 241/291 [00:04<00:00, 57.69it/s]
Loading 0:  86%|████████▌ | 250/291 [00:04<00:00, 61.80it/s]
Loading 0:  89%|████████▉ | 259/291 [00:05<00:00, 67.67it/s]
Loading 0:  92%|█████████▏| 268/291 [00:05<00:00, 71.72it/s]
Loading 0:  95%|█████████▌| 277/291 [00:05<00:00, 75.00it/s]
Loading 0:  98%|█████████▊| 286/291 [00:05<00:00, 78.29it/s]
                                                            

Job rinen0721-llama0929-v2-mkmlizer completed after 72.27s with status: succeeded
Stopping job with name rinen0721-llama0929-v2-mkmlizer
Pipeline stage MKMLizer completed in 73.05s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rinen0721-llama0929-v2
Waiting for inference service rinen0721-llama0929-v2 to be ready
rinen0721-llama0924-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rinen0721-llama0924-v2-mkmlizer: ║     _____            __           __                                ║
rinen0721-llama0924-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rinen0721-llama0924-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rinen0721-llama0924-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rinen0721-llama0924-v2-mkmlizer: ║       /___/                                                         ║
rinen0721-llama0924-v2-mkmlizer: ║                                                                     ║
rinen0721-llama0924-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
rinen0721-llama0924-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rinen0721-llama0924-v2-mkmlizer: ║  https://mk1.ai                                                     ║
rinen0721-llama0924-v2-mkmlizer: ║                                                                     ║
rinen0721-llama0924-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
rinen0721-llama0924-v2-mkmlizer: ║  belonging to:                                                      ║
rinen0721-llama0924-v2-mkmlizer: ║                                                                     ║
rinen0721-llama0924-v2-mkmlizer: ║  Chai Research Corp.                                                ║
rinen0721-llama0924-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rinen0721-llama0924-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rinen0721-llama0924-v2-mkmlizer: ║                                                                     ║
rinen0721-llama0924-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rinen0721-llama0924-v2-mkmlizer: Downloaded to shared memory in 23.226s
rinen0721-llama0924-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpfc_d8bzz, device:0
rinen0721-llama0924-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rinen0721-llama0924-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py:55: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
rinen0721-llama0924-v2-mkmlizer:   tensors = torch.load(model_shard_filename, map_location=torch.device(self.device), mmap=True)
rinen0721-llama0924-v2-mkmlizer: quantized model in 26.387s
rinen0721-llama0924-v2-mkmlizer: Processed model rinen0721/llama0924 in 49.614s
rinen0721-llama0924-v2-mkmlizer: creating bucket guanaco-mkml-models
rinen0721-llama0924-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rinen0721-llama0924-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rinen0721-llama0924-v2
rinen0721-llama0924-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rinen0721-llama0924-v2/config.json
rinen0721-llama0924-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rinen0721-llama0924-v2/special_tokens_map.json
rinen0721-llama0924-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rinen0721-llama0924-v2/tokenizer_config.json
rinen0721-llama0924-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rinen0721-llama0924-v2/tokenizer.json
rinen0721-llama0924-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rinen0721-llama0924-v2/flywheel_model.0.safetensors
rinen0721-llama0924-v2-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 6/291 [00:00<00:04, 59.77it/s]
Loading 0:   5%|▌         | 16/291 [00:00<00:04, 66.84it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:03, 73.25it/s]
Loading 0:  12%|█▏        | 34/291 [00:00<00:03, 74.86it/s]
Loading 0:  15%|█▍        | 43/291 [00:00<00:03, 79.22it/s]
Loading 0:  18%|█▊        | 52/291 [00:00<00:02, 82.04it/s]
Loading 0:  21%|██▏       | 62/291 [00:00<00:02, 87.04it/s]
Loading 0:  24%|██▍       | 71/291 [00:00<00:02, 85.69it/s]
Loading 0:  27%|██▋       | 80/291 [00:00<00:02, 83.65it/s]
Loading 0:  31%|███       | 89/291 [00:02<00:09, 20.28it/s]
Loading 0:  33%|███▎      | 97/291 [00:02<00:07, 25.41it/s]
Loading 0:  36%|███▋      | 106/291 [00:02<00:05, 32.13it/s]
Loading 0:  40%|███▉      | 115/291 [00:02<00:04, 39.45it/s]
Loading 0:  43%|████▎     | 124/291 [00:02<00:03, 47.04it/s]
Loading 0:  46%|████▌     | 133/291 [00:02<00:02, 53.52it/s]
Loading 0:  49%|████▉     | 142/291 [00:02<00:02, 58.31it/s]
Loading 0:  52%|█████▏    | 151/291 [00:02<00:02, 64.00it/s]
Loading 0:  55%|█████▍    | 160/291 [00:03<00:02, 64.94it/s]
Loading 0:  58%|█████▊    | 169/291 [00:03<00:01, 67.58it/s]
Loading 0:  61%|██████    | 178/291 [00:03<00:01, 71.93it/s]
Loading 0:  64%|██████▍   | 187/291 [00:04<00:04, 21.03it/s]
Loading 0:  67%|██████▋   | 196/291 [00:04<00:03, 26.83it/s]
Loading 0:  70%|███████   | 205/291 [00:04<00:02, 32.17it/s]
Loading 0:  74%|███████▎  | 214/291 [00:04<00:01, 38.95it/s]
Loading 0:  77%|███████▋  | 223/291 [00:04<00:01, 46.58it/s]
Loading 0:  80%|███████▉  | 232/291 [00:05<00:01, 52.11it/s]
Loading 0:  83%|████████▎ | 241/291 [00:05<00:00, 55.85it/s]
Loading 0:  86%|████████▌ | 250/291 [00:05<00:00, 62.16it/s]
Loading 0:  89%|████████▉ | 259/291 [00:05<00:00, 68.04it/s]
Loading 0:  92%|█████████▏| 268/291 [00:05<00:00, 73.12it/s]
Loading 0:  95%|█████████▌| 277/291 [00:05<00:00, 75.97it/s]
Loading 0:  98%|█████████▊| 286/291 [00:05<00:00, 78.60it/s]
                                                            

Job rinen0721-llama0924-v2-mkmlizer completed after 123.65s with status: succeeded
Stopping job with name rinen0721-llama0924-v2-mkmlizer
Pipeline stage MKMLizer completed in 124.23s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rinen0721-llama0924-v2
Waiting for inference service rinen0721-llama0924-v2 to be ready
Inference service rinen0721-llama0929-v2 ready after 140.58880496025085s
Pipeline stage MKMLDeployer completed in 141.01s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.780888557434082s
Received healthy response to inference request in 1.3117215633392334s
Received healthy response to inference request in 1.4018936157226562s
Received healthy response to inference request in 1.2753713130950928s
Received healthy response to inference request in 1.5025358200073242s
5 requests
0 failed requests
5th percentile: 1.2826413631439209
10th percentile: 1.289911413192749
20th percentile: 1.3044515132904053
30th percentile: 1.329755973815918
40th percentile: 1.3658247947692872
50th percentile: 1.4018936157226562
60th percentile: 1.4421504974365233
70th percentile: 1.4824073791503907
80th percentile: 1.5582063674926758
90th percentile: 1.6695474624633788
95th percentile: 1.7252180099487304
99th percentile: 1.7697544479370118
mean time: 1.4544821739196778
Pipeline stage StressChecker completed in 8.52s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.85s
Shutdown handler de-registered
rinen0721-llama0929_v2 status is now deployed due to DeploymentManager action
rinen0721-llama0929_v2 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8095da1e-7e11-453e-b4d8-198bf948d978, 6dc94916-cd4c-468f-bf76-abb9a4dbb81b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'11db9427-476a-4b37-89d8-c4ad9c5936e2, df2c9db5-9234-4a73-9b89-fcdb0fbd8cb6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cc664473-e22e-4efd-a2b1-ba24f24017a5, 7a63467f-4b66-4321-b33f-dd39dbfbe5d8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1ce0790c-5085-4857-9dfe-549f1b0d61f6, 326a05b8-6a07-400e-abf2-88fae2ef09c9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7842f6d1-e645-4bb1-ac4a-df7b32f8c727, 98596c7a-9dab-43f4-aa1b-c50a13f1dda0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:37:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'628c3b78-97fd-4e3c-9670-e84fccffea50, b7e67c3f-6dad-415c-87dc-df1ee66dd41f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4a27558f-85b3-443d-9f13-aaaa767818fb, e1dd68aa-2a0d-40ee-8fec-d6d6c24a66bf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b1dcf47e-c6e0-4bc8-ba15-1d343d62be2b, 0b972e5d-2475-41e5-8935-e1e5c8adcc80\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b8c3f235-c37e-414d-9c60-3b172881460c, bcac14ca-0f40-4761-9d25-3855124d49ee\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:19 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c5ac844d-0779-458d-bab5-db2e2079e168, 9c608d35-2df9-41e1-acaa-ee423ec54c82\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'04af4059-7881-4594-bed6-bac6ebb4e76f, 590f8db4-f43c-409f-8430-a539e04f8a45\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'34f555c9-61df-4b0e-8dd0-e5dbc4f994c3, 55aae0aa-8603-40dd-a742-802d4e1fe226\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'414acef3-f396-4849-9081-2e40c67d848a, 4d17cc2c-2fd7-4e86-bcf2-774029b1459f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a77d722e-f07c-47bc-959b-ea9336c7360f, 345d4158-392c-4808-aaac-cf475f2fb6f3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d194123a-d7b0-48b2-a5f5-268e1933e0f8, ec1a72f9-3d9e-4df7-829c-57e62cdc6d1c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d92d3622-3d29-4e9a-a3d7-28fc137ec12b, c14cf230-d382-4b9d-8b3d-dab9c038aeee\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:32 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9fbda67d-de6c-4802-8a5d-3d8b5c0925da, 7cd571a4-fd8e-4efb-8c60-1a1878fe5a51\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
admin requested tearing down of sao10k-12b-party-v1_v1
Checking if service rinen0721-llama0929-v2 is running
Shutdown handler not registered because Python interpreter is not running in the main thread
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'02b1f68b-a9c0-467d-afb1-8fd006bd698e, b27fdd89-7d83-42d3-a89c-526b2209a648\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:37 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
run pipeline %s
Shutdown handler de-registered
run pipeline stage %s
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'43221c3b-bf8e-43b1-bab7-4e2181ba7c25, 182589cb-7632-4892-bfdb-ea8a65320465\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:37 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ab7962eb-146d-4e9e-ac22-644f6471de51, 4b6b259a-b3db-42e7-83cd-bf3d1983893e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:38 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'94e4e050-4445-459d-8341-5a7b85393a8c, d1a63615-6b2f-4093-bb39-1cfcc458338d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'39747cb0-d753-4cf8-8d95-bf3afcf27871, 60474532-cc58-440f-a22f-5c64ab1c4849\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:54 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'824aece2-6e45-4d47-9f94-8fb728cab47a, b10e2589-ad6a-440a-be8e-343931f663df\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d09c7518-cee4-4381-892c-e46c7349b755, 1f8ea8bd-dbf0-452d-b1bc-6c21be265e9b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c547dd38-c9d6-452d-a268-8a8e5e1eab87, 8151e776-3d5e-4c08-b671-a20d71a5c3ce\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:42 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'499d542b-bbd9-4d1a-9dbc-ab6e9d238db7, be55286a-b04f-47c6-8830-a600778f3b37\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rinen0721-llama0929_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
%s, retrying in %s seconds...
Checking if service rinen0721-llama0929-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'852e77ea-d862-444b-a42d-87d7753c2528, 7dafb1df-fdaf-4435-95e2-96ac6d04bd6e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
rinen0721-llama0929_v2 status is now torndown due to DeploymentManager action