developer_uid: rica40325
submission_id: rica40325-mistral-test1_v5
model_name: rica40325-mistral-test1_v1
model_group: rica40325/mistral-test1
status: torndown
timestamp: 2024-10-01T06:36:17+00:00
num_battles: 3638
num_wins: 1998
celo_rating: 1286.9
family_friendly_score: 0.5403233475986686
family_friendly_standard_error: 0.008387848952993963
submission_type: basic
model_repo: rica40325/mistral-test1
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: rica40325-mistral-test1_v1
ineligible_reason: num_battles<5000
is_internal_developer: False
language_model: rica40325/mistral-test1
model_size: 13B
ranking_group: single
us_pacific_date: 2024-09-30
win_ratio: 0.5492028587135789
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rica40325-mistral-test1-v5-mkmlizer
Waiting for job on rica40325-mistral-test1-v5-mkmlizer to finish
rica40325-mistral-test1-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rica40325-mistral-test1-v5-mkmlizer: ║     _____            __           __                                ║
rica40325-mistral-test1-v5-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rica40325-mistral-test1-v5-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rica40325-mistral-test1-v5-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rica40325-mistral-test1-v5-mkmlizer: ║       /___/                                                         ║
rica40325-mistral-test1-v5-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v5-mkmlizer: ║  Version: 0.11.12                                                   ║
rica40325-mistral-test1-v5-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rica40325-mistral-test1-v5-mkmlizer: ║  https://mk1.ai                                                     ║
rica40325-mistral-test1-v5-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v5-mkmlizer: ║  The license key for the current software has been verified as      ║
rica40325-mistral-test1-v5-mkmlizer: ║  belonging to:                                                      ║
rica40325-mistral-test1-v5-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v5-mkmlizer: ║  Chai Research Corp.                                                ║
rica40325-mistral-test1-v5-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rica40325-mistral-test1-v5-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rica40325-mistral-test1-v5-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rica40325-mistral-test1-v5-mkmlizer: Downloaded to shared memory in 63.475s
rica40325-mistral-test1-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpxtr85mqj, device:0
rica40325-mistral-test1-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rica40325-mistral-test1-v5-mkmlizer: quantized model in 40.935s
rica40325-mistral-test1-v5-mkmlizer: Processed model rica40325/mistral-test1 in 104.410s
rica40325-mistral-test1-v5-mkmlizer: creating bucket guanaco-mkml-models
rica40325-mistral-test1-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rica40325-mistral-test1-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rica40325-mistral-test1-v5
rica40325-mistral-test1-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rica40325-mistral-test1-v5/config.json
rica40325-mistral-test1-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rica40325-mistral-test1-v5/special_tokens_map.json
rica40325-mistral-test1-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rica40325-mistral-test1-v5/tokenizer_config.json
rica40325-mistral-test1-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rica40325-mistral-test1-v5/flywheel_model.0.safetensors
rica40325-mistral-test1-v5-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:15, 23.52it/s]
Loading 0:   3%|▎         | 10/363 [00:00<00:12, 29.00it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:13, 25.48it/s]
Loading 0:   6%|▌         | 20/363 [00:00<00:09, 34.39it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:13, 24.22it/s]
Loading 0:   7%|▋         | 27/363 [00:01<00:14, 22.96it/s]
Loading 0:   9%|▊         | 31/363 [00:01<00:13, 24.27it/s]
Loading 0:   9%|▉         | 34/363 [00:01<00:15, 21.92it/s]
Loading 0:  10%|█         | 37/363 [00:01<00:15, 21.60it/s]
Loading 0:  11%|█         | 40/363 [00:01<00:14, 22.03it/s]
Loading 0:  12%|█▏        | 43/363 [00:01<00:15, 20.27it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:15, 20.32it/s]
Loading 0:  13%|█▎        | 49/363 [00:02<00:14, 21.06it/s]
Loading 0:  14%|█▍        | 52/363 [00:02<00:15, 19.61it/s]
Loading 0:  15%|█▌        | 55/363 [00:02<00:15, 20.26it/s]
Loading 0:  17%|█▋        | 60/363 [00:02<00:12, 24.71it/s]
Loading 0:  17%|█▋        | 63/363 [00:02<00:15, 19.11it/s]
Loading 0:  18%|█▊        | 66/363 [00:02<00:14, 20.52it/s]
Loading 0:  20%|█▉        | 71/363 [00:03<00:11, 24.64it/s]
Loading 0:  20%|██        | 74/363 [00:03<00:11, 25.69it/s]
Loading 0:  21%|██        | 77/363 [00:03<00:13, 21.75it/s]
Loading 0:  23%|██▎       | 84/363 [00:03<00:09, 29.30it/s]
Loading 0:  24%|██▍       | 88/363 [00:03<00:09, 28.54it/s]
Loading 0:  26%|██▌       | 93/363 [00:03<00:08, 30.74it/s]
Loading 0:  27%|██▋       | 97/363 [00:03<00:09, 29.41it/s]
Loading 0:  28%|██▊       | 101/363 [00:04<00:11, 23.22it/s]
Loading 0:  29%|██▊       | 104/363 [00:04<00:12, 21.04it/s]
Loading 0:  31%|███       | 111/363 [00:04<00:08, 28.01it/s]
Loading 0:  32%|███▏      | 115/363 [00:04<00:08, 27.59it/s]
Loading 0:  33%|███▎      | 120/363 [00:04<00:08, 30.25it/s]
Loading 0:  34%|███▍      | 124/363 [00:04<00:08, 29.27it/s]
Loading 0:  36%|███▌      | 129/363 [00:05<00:07, 31.50it/s]
Loading 0:  37%|███▋      | 133/363 [00:05<00:07, 30.18it/s]
Loading 0:  38%|███▊      | 137/363 [00:05<00:07, 30.03it/s]
Loading 0:  39%|███▉      | 141/363 [00:05<00:06, 32.16it/s]
Loading 0:  40%|███▉      | 145/363 [00:05<00:09, 22.53it/s]
Loading 0:  41%|████      | 149/363 [00:06<00:09, 22.18it/s]
Loading 0:  43%|████▎     | 156/363 [00:06<00:07, 29.10it/s]
Loading 0:  44%|████▍     | 160/363 [00:06<00:07, 28.93it/s]
Loading 0:  45%|████▌     | 165/363 [00:06<00:06, 31.29it/s]
Loading 0:  47%|████▋     | 169/363 [00:06<00:06, 30.38it/s]
Loading 0:  48%|████▊     | 174/363 [00:06<00:05, 32.03it/s]
Loading 0:  49%|████▉     | 178/363 [00:06<00:06, 29.90it/s]
Loading 0:  50%|█████     | 182/363 [00:07<00:07, 24.75it/s]
Loading 0:  51%|█████     | 185/363 [00:07<00:08, 21.85it/s]
Loading 0:  53%|█████▎    | 192/363 [00:07<00:05, 28.62it/s]
Loading 0:  54%|█████▍    | 196/363 [00:07<00:06, 27.51it/s]
Loading 0:  55%|█████▌    | 201/363 [00:07<00:05, 29.60it/s]
Loading 0:  56%|█████▋    | 205/363 [00:07<00:05, 28.61it/s]
Loading 0:  58%|█████▊    | 210/363 [00:08<00:04, 30.83it/s]
Loading 0:  59%|█████▉    | 214/363 [00:08<00:05, 29.32it/s]
Loading 0:  60%|██████    | 218/363 [00:08<00:04, 29.50it/s]
Loading 0:  61%|██████▏   | 223/363 [00:08<00:05, 25.21it/s]
Loading 0:  62%|██████▏   | 226/363 [00:08<00:05, 23.79it/s]
Loading 0:  63%|██████▎   | 230/363 [00:08<00:05, 22.55it/s]
Loading 0:  65%|██████▌   | 237/363 [00:09<00:04, 28.97it/s]
Loading 0:  66%|██████▋   | 241/363 [00:09<00:04, 28.14it/s]
Loading 0:  68%|██████▊   | 246/363 [00:09<00:03, 30.32it/s]
Loading 0:  69%|██████▉   | 250/363 [00:09<00:03, 28.94it/s]
Loading 0:  70%|███████   | 255/363 [00:09<00:03, 31.18it/s]
Loading 0:  71%|███████▏  | 259/363 [00:09<00:03, 29.92it/s]
Loading 0:  72%|███████▏  | 263/363 [00:10<00:04, 23.83it/s]
Loading 0:  73%|███████▎  | 266/363 [00:10<00:04, 21.32it/s]
Loading 0:  75%|███████▍  | 271/363 [00:10<00:03, 26.53it/s]
Loading 0:  76%|███████▌  | 275/363 [00:10<00:03, 24.52it/s]
Loading 0:  78%|███████▊  | 282/363 [00:10<00:02, 31.07it/s]
Loading 0:  79%|███████▉  | 286/363 [00:10<00:02, 29.71it/s]
Loading 0:  80%|████████  | 291/363 [00:10<00:02, 31.71it/s]
Loading 0:  81%|████████▏ | 295/363 [00:11<00:02, 30.15it/s]
Loading 0:  82%|████████▏ | 299/363 [00:11<00:02, 29.94it/s]
Loading 0:  84%|████████▎ | 304/363 [00:11<00:02, 25.65it/s]
Loading 0:  85%|████████▍ | 307/363 [00:11<00:02, 24.41it/s]
Loading 0:  86%|████████▌ | 311/363 [00:11<00:02, 23.47it/s]
Loading 0:  88%|████████▊ | 318/363 [00:11<00:01, 30.41it/s]
Loading 0:  89%|████████▊ | 322/363 [00:12<00:01, 29.48it/s]
Loading 0:  90%|█████████ | 327/363 [00:12<00:01, 31.91it/s]
Loading 0:  91%|█████████ | 331/363 [00:12<00:01, 30.65it/s]
Loading 0:  93%|█████████▎| 336/363 [00:12<00:00, 32.30it/s]
Loading 0:  94%|█████████▎| 340/363 [00:12<00:00, 30.66it/s]
Loading 0:  95%|█████████▍| 344/363 [00:19<00:09,  1.99it/s]
Loading 0:  96%|█████████▌| 348/363 [00:19<00:05,  2.67it/s]
Loading 0:  97%|█████████▋| 353/363 [00:19<00:02,  3.88it/s]
Loading 0:  98%|█████████▊| 357/363 [00:20<00:01,  5.03it/s]
                                                            

Job rica40325-mistral-test1-v5-mkmlizer completed after 133.54s with status: succeeded
Stopping job with name rica40325-mistral-test1-v5-mkmlizer
Pipeline stage MKMLizer completed in 134.39s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rica40325-mistral-test1-v5
Waiting for inference service rica40325-mistral-test1-v5 to be ready
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rica40325-mistral-test1-v6-mkmlizer
Waiting for job on rica40325-mistral-test1-v6-mkmlizer to finish
rica40325-mistral-test1-v6-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rica40325-mistral-test1-v6-mkmlizer: ║     _____            __           __                                ║
rica40325-mistral-test1-v6-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rica40325-mistral-test1-v6-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rica40325-mistral-test1-v6-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rica40325-mistral-test1-v6-mkmlizer: ║       /___/                                                         ║
rica40325-mistral-test1-v6-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v6-mkmlizer: ║  Version: 0.11.12                                                   ║
rica40325-mistral-test1-v6-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rica40325-mistral-test1-v6-mkmlizer: ║  https://mk1.ai                                                     ║
rica40325-mistral-test1-v6-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v6-mkmlizer: ║  The license key for the current software has been verified as      ║
rica40325-mistral-test1-v6-mkmlizer: ║  belonging to:                                                      ║
rica40325-mistral-test1-v6-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v6-mkmlizer: ║  Chai Research Corp.                                                ║
rica40325-mistral-test1-v6-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rica40325-mistral-test1-v6-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rica40325-mistral-test1-v6-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v6-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rica40325-mistral-test1-v7-mkmlizer
Waiting for job on rica40325-mistral-test1-v7-mkmlizer to finish
rica40325-mistral-test1-v7-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rica40325-mistral-test1-v7-mkmlizer: ║     _____            __           __                                ║
rica40325-mistral-test1-v7-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rica40325-mistral-test1-v7-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rica40325-mistral-test1-v7-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rica40325-mistral-test1-v7-mkmlizer: ║       /___/                                                         ║
rica40325-mistral-test1-v7-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v7-mkmlizer: ║  Version: 0.11.12                                                   ║
rica40325-mistral-test1-v7-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rica40325-mistral-test1-v7-mkmlizer: ║  https://mk1.ai                                                     ║
rica40325-mistral-test1-v7-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v7-mkmlizer: ║  The license key for the current software has been verified as      ║
rica40325-mistral-test1-v7-mkmlizer: ║  belonging to:                                                      ║
rica40325-mistral-test1-v7-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v7-mkmlizer: ║  Chai Research Corp.                                                ║
rica40325-mistral-test1-v7-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rica40325-mistral-test1-v7-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
rica40325-mistral-test1-v7-mkmlizer: ║                                                                     ║
rica40325-mistral-test1-v7-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rica40325-mistral-test1-v6-mkmlizer: Downloaded to shared memory in 58.923s
rica40325-mistral-test1-v6-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp19p6_g0a, device:0
rica40325-mistral-test1-v6-mkmlizer: Saving flywheel model at /dev/shm/model_cache
admin requested tearing down of cycy233-nemo-p-e-v1-c1_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of cycy233-nemo-p-e-v1-c2_v1
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of cycy233-nemo-p-e-v1-c3_v1
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of cycy233-nemo-p-e-v1-c4_v1
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of cycy233-nemo-p-e-v2-c1_v1
Running pipeline stage MKMLDeleter
Checking if service cycy233-nemo-p-e-v1-c1-v1 is running
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of cycy233-nemo-p-e-v2-c2_v1
Checking if service cycy233-nemo-p-e-v1-c2-v1 is running
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of cycy233-nemo-p-e-v2-c3_v1
Checking if service cycy233-nemo-p-e-v1-c3-v1 is running
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of cycy233-nemo-p-e-v2-c4_v1
Tearing down inference service cycy233-nemo-p-e-v1-c1-v1
Tearing down inference service cycy233-nemo-p-e-v1-c2-v1
Checking if service cycy233-nemo-p-e-v1-c4-v1 is running
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Service cycy233-nemo-p-e-v1-c1-v1 has been torndown
Service cycy233-nemo-p-e-v1-c2-v1 has been torndown
Checking if service cycy233-nemo-p-e-v2-c1-v1 is running
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline %s
Pipeline stage MKMLDeleter completed in 5.18s
Pipeline stage MKMLDeleter completed in 4.86s
Tearing down inference service cycy233-nemo-p-e-v1-c3-v1
Checking if service cycy233-nemo-p-e-v2-c2-v1 is running
Running pipeline stage MKMLDeleter
run pipeline stage %s
run pipeline stage %s
run pipeline stage %s
Tearing down inference service cycy233-nemo-p-e-v1-c4-v1
Service cycy233-nemo-p-e-v1-c3-v1 has been torndown
Checking if service cycy233-nemo-p-e-v2-c3-v1 is running
Running pipeline stage MKMLDeleter
Running pipeline stage MKMLModelDeleter
Tearing down inference service cycy233-nemo-p-e-v2-c1-v1
Running pipeline stage MKMLModelDeleter
Service cycy233-nemo-p-e-v1-c4-v1 has been torndown
Pipeline stage MKMLDeleter completed in 6.27s
Checking if service cycy233-nemo-p-e-v2-c4-v1 is running
Cleaning model data from S3
Cleaning model data from S3
Service cycy233-nemo-p-e-v2-c1-v1 has been torndown
Tearing down inference service cycy233-nemo-p-e-v2-c2-v1
Pipeline stage MKMLDeleter completed in 6.53s
run pipeline stage %s
Cleaning model data from model cache
rica40325-mistral-test1-v6-mkmlizer: quantized model in 40.610s
Cleaning model data from model cache
Tearing down inference service cycy233-nemo-p-e-v2-c3-v1
Pipeline stage MKMLDeleter completed in 6.35s
run pipeline stage %s
Service cycy233-nemo-p-e-v2-c2-v1 has been torndown
Running pipeline stage MKMLModelDeleter
rica40325-mistral-test1-v6-mkmlizer: Processed model rica40325/mistral-test1 in 99.533s
run pipeline stage %s
Service cycy233-nemo-p-e-v2-c3-v1 has been torndown
Running pipeline stage MKMLModelDeleter
Tearing down inference service cycy233-nemo-p-e-v2-c4-v1
Pipeline stage MKMLDeleter completed in 7.17s
Cleaning model data from S3
rica40325-mistral-test1-v6-mkmlizer: creating bucket guanaco-mkml-models
Running pipeline stage MKMLModelDeleter
Pipeline stage MKMLDeleter completed in 6.73s
Cleaning model data from S3
run pipeline stage %s
Service cycy233-nemo-p-e-v2-c4-v1 has been torndown
Cleaning model data from model cache
rica40325-mistral-test1-v6-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
Cleaning model data from S3
run pipeline stage %s
Cleaning model data from model cache
Running pipeline stage MKMLModelDeleter
Pipeline stage MKMLDeleter completed in 7.34s
rica40325-mistral-test1-v6-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rica40325-mistral-test1-v6
Cleaning model data from model cache
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
run pipeline stage %s
rica40325-mistral-test1-v6-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rica40325-mistral-test1-v6/config.json
Cleaning model data from S3
Cleaning model data from model cache
Running pipeline stage MKMLModelDeleter
rica40325-mistral-test1-v6-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rica40325-mistral-test1-v6/special_tokens_map.json
Cleaning model data from model cache
Cleaning model data from S3
rica40325-mistral-test1-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rica40325-mistral-test1-v6/tokenizer_config.json
Cleaning model data from model cache
rica40325-mistral-test1-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rica40325-mistral-test1-v6/tokenizer.json
Deleting key cycy233-nemo-p-e-v2-c4-v1/config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c2-v1/config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c3-v1/config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c3-v1/config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c1-v1/config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c1-v1/config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c4-v1/config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c2-v1/config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c4-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c2-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c3-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c3-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
rica40325-mistral-test1-v7-mkmlizer: Downloaded to shared memory in 65.551s
Deleting key cycy233-nemo-p-e-v1-c1-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c1-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c4-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c2-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
rica40325-mistral-test1-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpl4xou52_, device:0
Deleting key cycy233-nemo-p-e-v2-c4-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c2-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c3-v1/special_tokens_map.json from bucket guanaco-mkml-models
rica40325-mistral-test1-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Deleting key cycy233-nemo-p-e-v1-c3-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c1-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c4-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c1-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c4-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c2-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c2-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c3-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c3-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c1-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c4-v1/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c1-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c4-v1/tokenizer.json from bucket guanaco-mkml-models
Inference service rica40325-mistral-test1-v5 ready after 140.6842110157013s
Deleting key cycy233-nemo-p-e-v2-c2-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c2-v1/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c3-v1/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c3-v1/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v1-c1-v1/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key cycy233-nemo-p-e-v2-c1-v1/tokenizer_config.json from bucket guanaco-mkml-models
Pipeline stage MKMLModelDeleter completed in 11.53s
Deleting key cycy233-nemo-p-e-v1-c4-v1/tokenizer_config.json from bucket guanaco-mkml-models
Pipeline stage MKMLDeployer completed in 143.00s
Deleting key cycy233-nemo-p-e-v2-c2-v1/tokenizer_config.json from bucket guanaco-mkml-models
Pipeline stage MKMLModelDeleter completed in 21.85s
Pipeline stage MKMLModelDeleter completed in 14.42s
Pipeline stage MKMLModelDeleter completed in 19.55s
Pipeline stage MKMLModelDeleter completed in 22.68s
Shutdown handler de-registered
Pipeline stage MKMLModelDeleter completed in 17.98s
run pipeline stage %s
Job rica40325-mistral-test1-v6-mkmlizer completed after 134.97s with status: succeeded
Pipeline stage MKMLModelDeleter completed in 19.15s
Pipeline stage MKMLModelDeleter completed in 16.64s
Shutdown handler de-registered
Shutdown handler de-registered
Shutdown handler de-registered
Shutdown handler de-registered
Shutdown handler de-registered
cycy233-nemo-p-e-v2-c4_v1 status is now torndown due to DeploymentManager action
Running pipeline stage StressChecker
Stopping job with name rica40325-mistral-test1-v6-mkmlizer
Shutdown handler de-registered
Shutdown handler de-registered
cycy233-nemo-p-e-v1-c2_v1 status is now torndown due to DeploymentManager action
cycy233-nemo-p-e-v2-c3_v1 status is now torndown due to DeploymentManager action
cycy233-nemo-p-e-v1-c3_v1 status is now torndown due to DeploymentManager action
cycy233-nemo-p-e-v1-c1_v1 status is now torndown due to DeploymentManager action
cycy233-nemo-p-e-v2-c1_v1 status is now torndown due to DeploymentManager action
cycy233-nemo-p-e-v1-c4_v1 status is now torndown due to DeploymentManager action
Pipeline stage MKMLizer completed in 139.69s
cycy233-nemo-p-e-v2-c2_v1 status is now torndown due to DeploymentManager action
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rica40325-mistral-test1-v6
Waiting for inference service rica40325-mistral-test1-v6 to be ready
Received healthy response to inference request in 2.8224754333496094s
Received healthy response to inference request in 1.8455398082733154s
Received healthy response to inference request in 1.9423527717590332s
Received healthy response to inference request in 2.529998540878296s
Received healthy response to inference request in 1.8076255321502686s
5 requests
0 failed requests
5th percentile: 1.8152083873748779
10th percentile: 1.8227912425994872
20th percentile: 1.837956953048706
30th percentile: 1.864902400970459
40th percentile: 1.903627586364746
50th percentile: 1.9423527717590332
60th percentile: 2.1774110794067383
70th percentile: 2.4124693870544434
80th percentile: 2.5884939193725587
90th percentile: 2.705484676361084
95th percentile: 2.7639800548553466
99th percentile: 2.8107763576507567
mean time: 2.1895984172821046
Pipeline stage StressChecker completed in 16.05s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.75s
Shutdown handler de-registered
rica40325-mistral-test1_v5 status is now deployed due to DeploymentManager action
rica40325-mistral-test1_v5 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c92ac8f1-e2e1-4c78-ab96-fc86d496321d, c237fba9-dcfe-4f8e-a9d7-bc7c7a5242c3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a19d5716-35a1-4e12-8f91-0c63a84b253b, f74664d7-b66e-44a1-bc4a-0cb663e06d7d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:07 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4a8d689a-e039-4aaf-9692-78d5ae9b2d89, ca5c5e72-ea68-43ec-bb2b-389457c4cf07\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2aa6f949-22dc-49f8-9afd-ec7f5fa7bf60, 668ed026-aa54-424c-98ba-5d45a4df82fa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b2660cc8-2b2c-4f05-b397-6ceff644c5ed, fef8ee57-217e-4231-939d-36f390deb358\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:37:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4d04d030-759b-40a5-bd3a-f2ef52368671, 20017a6f-d0d5-4cdb-ad8b-8ae3c1424145\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'edacf4c7-d7bf-46fd-be0a-8f53a37763d9, 439334b1-3fc4-449e-814d-fc47861192da\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7a105fd9-e3a1-46d5-abd6-a3dab4faf5af, 22dc4f2e-c450-447d-8bec-9e9d225bc61b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'55f676bc-4028-4cc6-8104-052f9ac4aba9, 949750d9-9956-440b-8120-fa9ff5fed59e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fe659936-1e69-4434-8b29-2c91ace5b558, c8520730-96be-405d-801c-919c475084fa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5f22a8d5-3ba8-4a47-bd99-a2fc9b1c5e66, 1a80792f-9672-45ca-92be-f1acd4e9cab1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2df13c2b-d6a2-4f3c-8909-037c64324273, de2a97c4-7141-438e-9e6f-167a6d9efaf8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:23 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9ea565f9-dcf8-4be7-a7d7-9fed66d2b779, 77b43e3c-b6b9-4a59-9809-31043628f973\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'135db479-da80-4d6f-be48-54be81fcb2b1, 2000fb3e-08cc-444d-bb15-e1424665e7ac\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d5d75c21-e2a8-4beb-9692-683058b8088e, 30aa7b6f-b3bc-49b9-bc84-6aa323b326a3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'22e49753-c8b7-482f-b748-e7e8b83abc68, f64a5862-dba8-43d4-ad02-fd8f3547c619\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'504612fa-ea04-4165-991e-945ebab3cc3b, 777822d0-ce95-4c87-be9a-39c83c1e8a24\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7c467842-8563-4f92-a49e-9cda891bbb5e, 9694b5e8-72f6-4efc-9cf4-1579fb029962\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cf90c8c6-750d-440e-bbba-42332fc1fa22, a3d0ef39-d888-4d37-83a0-a1e653de2eb4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:34 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'70b104bd-7e75-4224-986c-4f750b533db3, 6ba7f20d-d08d-402c-87eb-9314bd252f35\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2cfeae1b-6076-40b8-b90f-f7b77e450a31, 8c4a808a-de45-47b2-bc6f-3f6d9fc8b7eb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:36 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5c41cab5-5eec-48c9-8ce0-68d47ce783a7, 7ec88c53-a091-4873-a38f-1143bc52c9ee\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test_v2
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dd2b4291-bb0f-4740-a7e9-93251deb3a53, 3e5b121a-e40e-487c-8bc8-a2d233ba6374\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ce6fc549-5b53-4590-9cc9-f03791c9bf4e, 5a698308-575b-4ff9-b29c-88f63481c65e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ab5d45d6-3193-4c00-9bf0-e6778f09886d, 0e880fa1-8a0d-406b-8859-b44b6a0a962b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c80fdc33-b3ed-4a17-98f6-415906324610, 3c155638-d3c3-4903-b524-920c1f13f487\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:42 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of rica40325-mistral-test1_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
%s, retrying in %s seconds...
Checking if service rica40325-mistral-test1-v5 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'574f2ba1-ce61-4260-b1da-60ff503e88cc, f772aa98-31e6-4025-8258-ae02def8a8dd\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
rica40325-mistral-test1_v5 status is now torndown due to DeploymentManager action