developer_uid: chace9580
submission_id: jic062-nemo-v1-8_v1
model_name: jic062-nemo-v1-8_v1
model_group: jic062/Nemo-v1.8
status: torndown
timestamp: 2024-09-29T01:19:59+00:00
num_battles: 7746
num_wins: 3801
celo_rating: 1247.94
family_friendly_score: 0.5945134434284154
family_friendly_standard_error: 0.00569742600163381
submission_type: basic
model_repo: jic062/Nemo-v1.8
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: jic062-nemo-v1-8_v1
is_internal_developer: False
language_model: jic062/Nemo-v1.8
model_size: 13B
ranking_group: single
us_pacific_date: 2024-09-28
win_ratio: 0.4907048799380325
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '/s', '<|im_end|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-nemo-v1-8-v1-mkmlizer
Waiting for job on jic062-nemo-v1-8-v1-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-nemo-v1-8-v2-mkmlizer
Waiting for job on jic062-nemo-v1-8-v2-mkmlizer to finish
jic062-nemo-v1-8-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-nemo-v1-8-v1-mkmlizer: ║ _____ __ __ ║
jic062-nemo-v1-8-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
jic062-nemo-v1-8-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
jic062-nemo-v1-8-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
jic062-nemo-v1-8-v1-mkmlizer: ║ /___/ ║
jic062-nemo-v1-8-v1-mkmlizer: ║ ║
jic062-nemo-v1-8-v1-mkmlizer: ║ Version: 0.11.12 ║
jic062-nemo-v1-8-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
jic062-nemo-v1-8-v1-mkmlizer: ║ https://mk1.ai ║
jic062-nemo-v1-8-v1-mkmlizer: ║ ║
jic062-nemo-v1-8-v1-mkmlizer: ║ The license key for the current software has been verified as ║
jic062-nemo-v1-8-v1-mkmlizer: ║ belonging to: ║
jic062-nemo-v1-8-v1-mkmlizer: ║ ║
jic062-nemo-v1-8-v1-mkmlizer: ║ Chai Research Corp. ║
jic062-nemo-v1-8-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
jic062-nemo-v1-8-v1-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
jic062-nemo-v1-8-v1-mkmlizer: ║ ║
jic062-nemo-v1-8-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
jic062-nemo-v1-8-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-nemo-v1-8-v2-mkmlizer: ║ _____ __ __ ║
jic062-nemo-v1-8-v2-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
jic062-nemo-v1-8-v2-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
jic062-nemo-v1-8-v2-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
jic062-nemo-v1-8-v2-mkmlizer: ║ /___/ ║
jic062-nemo-v1-8-v2-mkmlizer: ║ ║
jic062-nemo-v1-8-v2-mkmlizer: ║ Version: 0.11.12 ║
jic062-nemo-v1-8-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
jic062-nemo-v1-8-v2-mkmlizer: ║ https://mk1.ai ║
jic062-nemo-v1-8-v2-mkmlizer: ║ ║
jic062-nemo-v1-8-v2-mkmlizer: ║ The license key for the current software has been verified as ║
jic062-nemo-v1-8-v2-mkmlizer: ║ belonging to: ║
jic062-nemo-v1-8-v2-mkmlizer: ║ ║
jic062-nemo-v1-8-v2-mkmlizer: ║ Chai Research Corp. ║
jic062-nemo-v1-8-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
jic062-nemo-v1-8-v2-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
jic062-nemo-v1-8-v2-mkmlizer: ║ ║
jic062-nemo-v1-8-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
jic062-nemo-v1-8-v1-mkmlizer: Downloaded to shared memory in 56.379s
jic062-nemo-v1-8-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmps9jpjxi5, device:0
jic062-nemo-v1-8-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
jic062-nemo-v1-8-v2-mkmlizer: Downloaded to shared memory in 45.063s
jic062-nemo-v1-8-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpwb0rgvsw, device:0
jic062-nemo-v1-8-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
jic062-nemo-v1-8-v2-mkmlizer: quantized model in 38.316s
jic062-nemo-v1-8-v2-mkmlizer: Processed model jic062/Nemo-v1.8 in 83.380s
jic062-nemo-v1-8-v2-mkmlizer: creating bucket guanaco-mkml-models
jic062-nemo-v1-8-v1-mkmlizer: quantized model in 37.967s
jic062-nemo-v1-8-v1-mkmlizer: Processed model jic062/Nemo-v1.8 in 94.346s
jic062-nemo-v1-8-v1-mkmlizer: creating bucket guanaco-mkml-models
jic062-nemo-v1-8-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-nemo-v1-8-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-nemo-v1-8-v1
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/config.json
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/special_tokens_map.json
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/tokenizer_config.json
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/tokenizer.json
jic062-nemo-v1-8-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-nemo-v1-8-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-nemo-v1-8-v2
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/config.json
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/special_tokens_map.json
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/tokenizer_config.json
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/tokenizer.json
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/flywheel_model.0.safetensors
jic062-nemo-v1-8-v2-mkmlizer: Loading 0: 0%| | 0/363 [00:00<?, ?it/s] Loading 0: 1%|▏ | 5/363 [00:00<00:12, 27.99it/s] Loading 0: 3%|▎ | 12/363 [00:00<00:07, 45.58it/s] Loading 0: 5%|▍ | 18/363 [00:00<00:07, 44.70it/s] Loading 0: 6%|▋ | 23/363 [00:00<00:09, 34.05it/s] Loading 0: 8%|▊ | 30/363 [00:00<00:07, 42.58it/s] Loading 0: 10%|▉ | 35/363 [00:00<00:07, 41.30it/s] Loading 0: 11%|█ | 40/363 [00:00<00:07, 41.40it/s] Loading 0: 12%|█▏ | 45/363 [00:01<00:07, 41.62it/s] Loading 0: 14%|█▍ | 50/363 [00:01<00:09, 32.66it/s] Loading 0: 15%|█▌ | 56/363 [00:01<00:08, 37.13it/s] Loading 0: 17%|█▋ | 61/363 [00:01<00:10, 27.50it/s] Loading 0: 18%|█▊ | 65/363 [00:01<00:10, 27.31it/s] Loading 0: 20%|█▉ | 71/363 [00:02<00:08, 32.88it/s] Loading 0: 21%|██ | 75/363 [00:02<00:08, 32.66it/s] Loading 0: 22%|██▏ | 80/363 [00:02<00:08, 34.76it/s] Loading 0: 23%|██▎ | 84/363 [00:02<00:08, 33.63it/s] Loading 0: 25%|██▍ | 89/363 [00:02<00:07, 36.21it/s] Loading 0: 26%|██▌ | 93/363 [00:02<00:07, 34.10it/s] Loading 0: 27%|██▋ | 98/363 [00:02<00:07, 36.74it/s] Loading 0: 28%|██▊ | 102/363 [00:02<00:07, 35.05it/s] Loading 0: 29%|██▉ | 106/363 [00:02<00:07, 35.67it/s] Loading 0: 31%|███ | 112/363 [00:03<00:06, 39.88it/s] Loading 0: 32%|███▏ | 117/363 [00:03<00:06, 37.95it/s] Loading 0: 33%|███▎ | 121/363 [00:03<00:06, 37.09it/s] Loading 0: 34%|███▍ | 125/363 [00:03<00:06, 36.95it/s] Loading 0: 36%|███▌ | 129/363 [00:03<00:06, 35.06it/s] Loading 0: 37%|███▋ | 134/363 [00:03<00:06, 37.34it/s] Loading 0: 38%|███▊ | 138/363 [00:03<00:06, 34.63it/s] Loading 0: 39%|███▉ | 142/363 [00:04<00:08, 24.69it/s] Loading 0: 40%|███▉ | 145/363 [00:04<00:08, 24.62it/s] Loading 0: 41%|████ | 149/363 [00:04<00:08, 25.01it/s] Loading 0: 43%|████▎ | 156/363 [00:04<00:06, 33.03it/s] Loading 0: 44%|████▍ | 160/363 [00:04<00:06, 32.18it/s] Loading 0: 45%|████▌ | 165/363 [00:04<00:05, 33.18it/s] Loading 0: 47%|████▋ | 169/363 [00:04<00:05, 32.55it/s] Loading 0: 48%|████▊ | 174/363 [00:05<00:05, 35.42it/s] Loading 0: 49%|████▉ | 178/363 [00:05<00:05, 33.69it/s] Loading 0: 50%|█████ | 183/363 [00:05<00:05, 35.66it/s] Loading 0: 52%|█████▏ | 187/363 [00:05<00:05, 34.39it/s] Loading 0: 53%|█████▎ | 192/363 [00:05<00:04, 37.09it/s] Loading 0: 54%|█████▍ | 196/363 [00:05<00:04, 35.09it/s] Loading 0: 55%|█████▌ | 201/363 [00:05<00:04, 36.44it/s] Loading 0: 56%|█████▋ | 205/363 [00:05<00:04, 34.36it/s] Loading 0: 58%|█████▊ | 210/363 [00:06<00:04, 37.33it/s] Loading 0: 59%|█████▉ | 214/363 [00:06<00:04, 35.13it/s] Loading 0: 60%|██████ | 218/363 [00:06<00:04, 33.85it/s] Loading 0: 61%|██████▏ | 223/363 [00:06<00:05, 26.17it/s] Loading 0: 62%|██████▏ | 226/363 [00:06<00:05, 25.59it/s] Loading 0: 63%|██████▎ | 230/363 [00:06<00:05, 25.91it/s] Loading 0: 65%|██████▌ | 237/363 [00:07<00:03, 33.60it/s] Loading 0: 66%|██████▋ | 241/363 [00:07<00:03, 32.83it/s] Loading 0: 68%|██████▊ | 246/363 [00:07<00:03, 35.89it/s] Loading 0: 69%|██████▉ | 250/363 [00:07<00:03, 34.71it/s] Loading 0: 70%|███████ | 255/363 [00:07<00:02, 36.93it/s] Loading 0: 71%|███████▏ | 259/363 [00:07<00:02, 35.17it/s] Loading 0: 73%|███████▎ | 264/363 [00:07<00:02, 37.58it/s] Loading 0: 74%|███████▍ | 268/363 [00:07<00:02, 35.89it/s] Loading 0: 75%|███████▌ | 273/363 [00:07<00:02, 37.96it/s] Loading 0: 76%|███████▋ | 277/363 [00:08<00:02, 35.36it/s] Loading 0: 78%|███████▊ | 282/363 [00:08<00:02, 37.71it/s] Loading 0: 79%|███████▉ | 286/363 [00:08<00:02, 36.01it/s] Loading 0: 80%|████████ | 291/363 [00:08<00:01, 38.39it/s] Loading 0: 81%|████████▏ | 295/363 [00:08<00:01, 36.43it/s] Loading 0: 82%|████████▏ | 299/363 [00:08<00:01, 36.29it/s] Loading 0: 84%|████████▎ | 304/363 [00:15<00:27, 2.11it/s] Loading 0: 85%|████████▍ | 307/363 [00:15<00:21, 2.64it/s] Loading 0: 86%|████████▌ | 312/363 [00:15<00:13, 3.88it/s] Loading 0: 88%|████████▊ | 319/363 [00:15<00:06, 6.31it/s] Loading 0: 89%|████████▉ | 323/363 [00:16<00:05, 7.92it/s] Loading 0: 90%|█████████ | 328/363 [00:16<00:03, 10.69it/s] Loading 0: 92%|█████████▏| 333/363 [00:16<00:02, 13.76it/s] Loading 0: 93%|█████████▎| 338/363 [00:16<00:01, 17.03it/s] Loading 0: 94%|█████████▍| 342/363 [00:16<00:01, 19.75it/s] Loading 0: 95%|█████████▌| 346/363 [00:16<00:00, 22.84it/s] Loading 0: 96%|█████████▋| 350/363 [00:16<00:00, 24.84it/s] Loading 0: 98%|█████████▊| 355/363 [00:16<00:00, 29.65it/s] Loading 0: 99%|█████████▉| 360/363 [00:16<00:00, 31.95it/s]
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/flywheel_model.0.safetensors
jic062-nemo-v1-8-v1-mkmlizer: Loading 0: 0%| | 0/363 [00:00<?, ?it/s] Loading 0: 1%|▏ | 5/363 [00:00<00:12, 28.32it/s] Loading 0: 4%|▎ | 13/363 [00:00<00:07, 46.68it/s] Loading 0: 5%|▍ | 18/363 [00:00<00:07, 45.92it/s] Loading 0: 6%|▋ | 23/363 [00:00<00:09, 34.74it/s] Loading 0: 8%|▊ | 30/363 [00:00<00:07, 43.31it/s] Loading 0: 10%|▉ | 35/363 [00:00<00:07, 41.76it/s] Loading 0: 11%|█ | 40/363 [00:00<00:07, 41.60it/s] Loading 0: 12%|█▏ | 45/363 [00:01<00:07, 41.90it/s] Loading 0: 14%|█▍ | 50/363 [00:01<00:09, 32.68it/s] Loading 0: 15%|█▌ | 56/363 [00:01<00:08, 37.21it/s] Loading 0: 17%|█▋ | 61/363 [00:01<00:11, 25.88it/s] Loading 0: 18%|█▊ | 65/363 [00:01<00:11, 26.32it/s] Loading 0: 20%|█▉ | 71/363 [00:02<00:09, 32.21it/s] Loading 0: 21%|██ | 75/363 [00:02<00:09, 31.56it/s] Loading 0: 22%|██▏ | 80/363 [00:02<00:08, 35.28it/s] Loading 0: 23%|██▎ | 84/363 [00:02<00:08, 34.46it/s] Loading 0: 25%|██▍ | 89/363 [00:02<00:07, 37.54it/s] Loading 0: 26%|██▌ | 94/363 [00:02<00:07, 37.40it/s] Loading 0: 27%|██▋ | 98/363 [00:02<00:07, 37.18it/s] Loading 0: 28%|██▊ | 102/363 [00:02<00:07, 35.52it/s] Loading 0: 29%|██▉ | 106/363 [00:02<00:07, 36.32it/s] Loading 0: 31%|███ | 112/363 [00:03<00:06, 39.85it/s] Loading 0: 32%|███▏ | 117/363 [00:03<00:06, 37.34it/s] Loading 0: 33%|███▎ | 121/363 [00:03<00:06, 37.22it/s] Loading 0: 34%|███▍ | 125/363 [00:03<00:06, 37.55it/s] Loading 0: 36%|███▌ | 129/363 [00:03<00:06, 35.70it/s] Loading 0: 37%|███▋ | 134/363 [00:03<00:06, 38.01it/s] Loading 0: 38%|███▊ | 138/363 [00:03<00:06, 35.17it/s] Loading 0: 39%|███▉ | 142/363 [00:04<00:09, 23.79it/s] Loading 0: 40%|███▉ | 145/363 [00:04<00:09, 24.14it/s] Loading 0: 41%|████ | 149/363 [00:04<00:08, 24.92it/s] Loading 0: 43%|████▎ | 156/363 [00:04<00:06, 33.38it/s] Loading 0: 44%|████▍ | 160/363 [00:04<00:06, 31.60it/s] Loading 0: 45%|████▌ | 165/363 [00:04<00:05, 33.82it/s] Loading 0: 47%|████▋ | 169/363 [00:04<00:05, 33.18it/s] Loading 0: 48%|████▊ | 174/363 [00:05<00:05, 35.97it/s] Loading 0: 49%|████▉ | 178/363 [00:05<00:05, 34.30it/s] Loading 0: 50%|█████ | 183/363 [00:05<00:04, 36.08it/s] Loading 0: 52%|█████▏ | 187/363 [00:05<00:05, 34.77it/s] Loading 0: 53%|█████▎ | 192/363 [00:05<00:04, 37.38it/s] Loading 0: 54%|█████▍ | 196/363 [00:05<00:04, 35.12it/s] Loading 0: 55%|█████▌ | 201/363 [00:05<00:04, 36.28it/s] Loading 0: 56%|█████▋ | 205/363 [00:05<00:04, 34.84it/s] Loading 0: 58%|█████▊ | 210/363 [00:06<00:04, 37.83it/s] Loading 0: 59%|█████▉ | 214/363 [00:06<00:04, 36.20it/s] Loading 0: 60%|██████ | 218/363 [00:06<00:04, 35.73it/s] Loading 0: 61%|██████ | 222/363 [00:06<00:03, 36.45it/s] Loading 0: 62%|██████▏ | 226/363 [00:06<00:05, 23.38it/s] Loading 0: 63%|██████▎ | 230/363 [00:06<00:05, 24.25it/s] Loading 0: 65%|██████▌ | 237/363 [00:06<00:03, 32.13it/s] Loading 0: 66%|██████▋ | 241/363 [00:07<00:03, 31.99it/s] Loading 0: 68%|██████▊ | 246/363 [00:07<00:03, 35.24it/s] Loading 0: 69%|██████▉ | 250/363 [00:07<00:03, 34.18it/s] Loading 0: 70%|███████ | 255/363 [00:07<00:02, 37.07it/s] Loading 0: 71%|███████▏ | 259/363 [00:07<00:02, 35.12it/s] Loading 0: 73%|███████▎ | 264/363 [00:07<00:02, 37.33it/s] Loading 0: 74%|███████▍ | 268/363 [00:07<00:02, 35.58it/s] Loading 0: 75%|███████▌ | 273/363 [00:07<00:02, 36.49it/s] Loading 0: 76%|███████▋ | 277/363 [00:08<00:02, 35.18it/s] Loading 0: 78%|███████▊ | 282/363 [00:08<00:02, 38.35it/s] Loading 0: 79%|███████▉ | 286/363 [00:08<00:02, 36.74it/s] Loading 0: 80%|████████ | 291/363 [00:08<00:01, 38.73it/s] Loading 0: 81%|████████▏ | 295/363 [00:08<00:01, 36.54it/s] Loading 0: 82%|████████▏ | 299/363 [00:08<00:01, 36.54it/s] Loading 0: 84%|████████▎ | 304/363 [00:15<00:28, 2.10it/s] Loading 0: 85%|████████▍ | 307/363 [00:15<00:21, 2.64it/s] Loading 0: 86%|████████▌ | 312/363 [00:15<00:13, 3.88it/s] Loading 0: 88%|████████▊ | 319/363 [00:15<00:06, 6.32it/s] Loading 0: 89%|████████▉ | 324/363 [00:16<00:04, 8.40it/s] Loading 0: 91%|█████████ | 329/363 [00:16<00:03, 11.03it/s] Loading 0: 92%|█████████▏| 334/363 [00:16<00:02, 14.22it/s] Loading 0: 93%|█████████▎| 339/363 [00:16<00:01, 16.08it/s] Loading 0: 95%|█████████▌| 346/363 [00:16<00:00, 22.35it/s] Loading 0: 97%|█████████▋| 351/363 [00:16<00:00, 25.40it/s] Loading 0: 98%|█████████▊| 356/363 [00:16<00:00, 28.74it/s] Loading 0: 99%|█████████▉| 361/363 [00:16<00:00, 31.29it/s]
Job jic062-nemo-v1-8-v1-mkmlizer completed after 124.64s with status: succeeded
Stopping job with name jic062-nemo-v1-8-v1-mkmlizer
Pipeline stage MKMLizer completed in 125.51s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service jic062-nemo-v1-8-v1
Waiting for inference service jic062-nemo-v1-8-v1 to be ready
Job jic062-nemo-v1-8-v2-mkmlizer completed after 113.91s with status: succeeded
Stopping job with name jic062-nemo-v1-8-v2-mkmlizer
Pipeline stage MKMLizer completed in 114.30s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service jic062-nemo-v1-8-v2
Waiting for inference service jic062-nemo-v1-8-v2 to be ready
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-nemo-v1-6-v4-mkmlizer
Waiting for job on jic062-nemo-v1-6-v4-mkmlizer to finish
jic062-nemo-v1-6-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-nemo-v1-6-v4-mkmlizer: ║ _____ __ __ ║
jic062-nemo-v1-6-v4-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
jic062-nemo-v1-6-v4-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
jic062-nemo-v1-6-v4-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
jic062-nemo-v1-6-v4-mkmlizer: ║ /___/ ║
jic062-nemo-v1-6-v4-mkmlizer: ║ ║
jic062-nemo-v1-6-v4-mkmlizer: ║ Version: 0.11.12 ║
jic062-nemo-v1-6-v4-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
jic062-nemo-v1-6-v4-mkmlizer: ║ https://mk1.ai ║
jic062-nemo-v1-6-v4-mkmlizer: ║ ║
jic062-nemo-v1-6-v4-mkmlizer: ║ The license key for the current software has been verified as ║
jic062-nemo-v1-6-v4-mkmlizer: ║ belonging to: ║
jic062-nemo-v1-6-v4-mkmlizer: ║ ║
jic062-nemo-v1-6-v4-mkmlizer: ║ Chai Research Corp. ║
jic062-nemo-v1-6-v4-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
jic062-nemo-v1-6-v4-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
jic062-nemo-v1-6-v4-mkmlizer: ║ ║
jic062-nemo-v1-6-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Inference service jic062-nemo-v1-8-v1 ready after 230.64079999923706s
Pipeline stage MKMLDeployer completed in 231.16s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3941752910614014s
Received healthy response to inference request in 1.8138670921325684s
Inference service jic062-nemo-v1-8-v2 ready after 230.6058521270752s
Received healthy response to inference request in 2.036547899246216s
Pipeline stage MKMLDeployer completed in 231.05s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.058387279510498s
Received healthy response to inference request in 2.402942180633545s
Received healthy response to inference request in 1.9270868301391602s
5 requests
0 failed requests
5th percentile: 1.8365110397338866
10th percentile: 1.8591549873352051
Received healthy response to inference request in 1.7296473979949951s
20th percentile: 1.9044428825378419
30th percentile: 1.9489790439605712
40th percentile: 1.9927634716033935
50th percentile: 2.036547899246216
60th percentile: 2.045283651351929
70th percentile: 2.054019403457642
80th percentile: 2.125544881820679
90th percentile: 2.25986008644104
95th percentile: 2.3270176887512206
99th percentile: 2.380743770599365
mean time: 2.0460128784179688
Pipeline stage StressChecker completed in 12.39s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
jic062-nemo-v1-6-v4-mkmlizer: Downloaded to shared memory in 62.361s
jic062-nemo-v1-6-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpuqdz21op, device:0
jic062-nemo-v1-6-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Received healthy response to inference request in 1.7409923076629639s
Received healthy response to inference request in 1.69749116897583s
Received healthy response to inference request in 1.7477772235870361s
5 requests
0 failed requests
5th percentile: 1.703922414779663
10th percentile: 1.7103536605834961
20th percentile: 1.7232161521911622
30th percentile: 1.7319163799285888
40th percentile: 1.7364543437957765
50th percentile: 1.7409923076629639
60th percentile: 1.7437062740325928
70th percentile: 1.7464202404022218
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.44s
80th percentile: 1.878810214996338
Shutdown handler de-registered
90th percentile: 2.1408761978149413
90th percentile: 2.1408761978149413
jic062-nemo-v1-8_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service jic062-nemo-v1-8-v1-profiler
Waiting for inference service jic062-nemo-v1-8-v1-profiler to be ready
Tearing down inference service jic062-nemo-v1-8-v1-profiler
%s, retrying in %s seconds...
Creating inference service jic062-nemo-v1-8-v1-profiler
Waiting for inference service jic062-nemo-v1-8-v1-profiler to be ready
Tearing down inference service jic062-nemo-v1-8-v1-profiler
%s, retrying in %s seconds...
Creating inference service jic062-nemo-v1-8-v1-profiler
Waiting for inference service jic062-nemo-v1-8-v1-profiler to be ready
Tearing down inference service jic062-nemo-v1-8-v1-profiler
clean up pipeline due to error=%s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
Shutdown handler de-registered
jic062-nemo-v1-8_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9dbf2a63-f922-42c0-84c8-296745485013, b9ca982b-1492-4a0d-96a1-24d2111be985\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Checking if service jic062-dpo-v2-5-nemo-v1 is running
Shutdown handler not registered because Python interpreter is not running in the main thread
%s, retrying in %s seconds...
run pipeline %s
Checking if service jic062-dpo-v2-5-nemo-v1 is running
run pipeline stage %s
%s, retrying in %s seconds...
Running pipeline stage MKMLDeleter
Checking if service jic062-dpo-v2-5-nemo-v1 is running
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'25fbae8b-8021-4fe3-838d-e7d8805a82f6, 2b554dff-ee93-49e4-b1e6-353817f16bfa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
%s, retrying in %s seconds...
Shutdown handler de-registered
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'96a5cf68-48d2-476c-8b50-10684d3107eb, 2811fbd2-973d-47ac-91e2-49a4f91bd46f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d953062e-1a19-49fb-8eaf-59bab171f374, 51788bdc-9179-406e-81d6-af88f6082310\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:52 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e0bc255e-6f13-4b36-a244-94c1331e6977, 5d2df11c-33c0-425c-9f6c-3026b137fea5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7372782d-66f1-499a-9729-ff550e10a254, 073db418-6309-4a98-8c06-fa98796c3bf6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:35:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'414f20d2-c257-4b80-bda0-7ebd25036886, 98b12511-bcfd-4ef5-837d-b60eb8eb601e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:40:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5e0d715f-ca7d-48c7-a06c-2528b9811578, 4d45fdde-7469-4183-9ccc-2d1bf0b96549\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:45:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'828f3c6c-9caf-428e-b697-5a41ac7cf0d1, 94ffb2ad-dfdb-4aa8-b74d-ae778900123b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:50:57 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fd16b3d5-2185-4047-8535-8731aa18623a, b3a35fb8-20f2-4a20-a04b-16d936ea9eb6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:55:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c15d7d82-08ac-45d7-9564-2a98421e90c9, f250ae2e-031e-419e-b63a-8137f940fba3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'03b07762-1022-4329-9cbe-e6c7c0dd2203, f00c779d-bcba-4db3-babe-7ab302a48d7a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'97013621-df42-4f67-854d-2bf9cfdf52b2, f29a5c8b-5623-4074-b382-0f5af10e5800\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:02 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7ff87023-8c7c-4b34-ad6c-5a0e830239e9, 81b12205-9f04-4e54-bf70-29cf28a644af\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'335a580e-2a74-4a2a-803b-9d9770a9d2ff, 30f79813-8658-43e3-a5c2-43091820bbfd\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'97c66dc6-dca5-469a-8034-047c3c202cf7, 7b3baf94-ff2c-4aff-b415-0e579256355d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'86f3fb69-6137-459c-8978-a916f774f1f0, c79b81f5-9509-4da5-9bef-c14345f5e822\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
admin requested tearing down of jic062-nemo-v1-9_v7
%s, retrying in %s seconds...
Shutdown handler not registered because Python interpreter is not running in the main thread
Checking if service jic062-nemo-v1-8-v1 is running
run pipeline %s
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a93ba586-5034-4c77-9475-bccf2adab235, c9cc2df8-0435-44cb-8cae-829a48135ff0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
run pipeline stage %s
Shutdown handler de-registered
Running pipeline stage MKMLDeleter
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'168f9cde-98a1-452d-a6f9-cb124de22692, 3c5b11bd-1835-4ceb-9f57-95949453bb1a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a8655443-4ff2-4c88-87df-e42503bbb67a, 22ffde69-33d9-48ce-a34d-e5badce3f34c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c0a91a1b-c748-46f0-ab52-9718fcc62590, 46e924c4-f087-4e80-8863-347483930ef3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dc9edd43-d10b-47cb-a857-b11ea6425f61, 98acff03-7f74-4045-a72e-8a51de3ab305\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9836f5da-65e3-4062-93c7-3a7343581ae3, 55da2bfa-463c-415a-89eb-c8dd3870730d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1f9ae097-beef-48cb-89e2-1a6fc1102f0b, 8dafb80a-6103-4e68-9bd5-b20511d7fd76\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'be41a3e0-4a3e-4a3a-b3a3-3f945281acd1, da4e6012-4ee4-4bdf-b426-5860fabd1769\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a36f6523-8086-4c21-b1ae-4c33f95cac31, 71781fc1-fb11-41fe-9fd2-c5117c16866c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'928dbdf5-c5b2-4c55-80c2-8ad799f7e680, 973b3268-07b7-4549-b612-6cf3c4031437\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0ce629dc-ba6c-42bd-bba2-02b515c01e90, 18ce91c5-83c0-4fd5-b04a-ec8135ba4387\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
jic062-nemo-v1-8_v1 status is now torndown due to DeploymentManager action