jic062-nemo-v1-8

developer_uid: chace9580
submission_id: jic062-nemo-v1-8_v2
model_name: jic062-nemo-v1-8_v2
model_group: jic062/Nemo-v1.8
status: torndown
timestamp: 2024-09-29T01:20:18+00:00
num_battles: 8011
num_wins: 4124
celo_rating: 1263.0
family_friendly_score: 0.5933693883832388
family_friendly_standard_error: 0.005606609386002264
submission_type: basic
model_repo: jic062/Nemo-v1.8
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: jic062-nemo-v1-8_v2
is_internal_developer: False
language_model: jic062/Nemo-v1.8
model_size: 13B
ranking_group: single
us_pacific_date: 2024-09-28
win_ratio: 0.514792160778929
generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '/s', '[/INST]'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '[INST]system\n{memory}[/INST]\n', 'prompt_template': '[INST]user\n{prompt}[/INST]\n', 'bot_template': '[INST]assistant\n{bot_name}: {message}[/INST]\n', 'user_template': '[INST]user\n{user_name}: {message}[/INST]\n', 'response_template': '[INST]assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-nemo-v1-8-v2-mkmlizer
Waiting for job on jic062-nemo-v1-8-v2-mkmlizer to finish
jic062-nemo-v1-8-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-nemo-v1-8-v1-mkmlizer: ║     _____            __           __                                ║
jic062-nemo-v1-8-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
jic062-nemo-v1-8-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
jic062-nemo-v1-8-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
jic062-nemo-v1-8-v1-mkmlizer: ║       /___/                                                         ║
jic062-nemo-v1-8-v1-mkmlizer: ║                                                                     ║
jic062-nemo-v1-8-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
jic062-nemo-v1-8-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
jic062-nemo-v1-8-v1-mkmlizer: ║  https://mk1.ai                                                     ║
jic062-nemo-v1-8-v1-mkmlizer: ║                                                                     ║
jic062-nemo-v1-8-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
jic062-nemo-v1-8-v1-mkmlizer: ║  belonging to:                                                      ║
jic062-nemo-v1-8-v1-mkmlizer: ║                                                                     ║
jic062-nemo-v1-8-v1-mkmlizer: ║  Chai Research Corp.                                                ║
jic062-nemo-v1-8-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
jic062-nemo-v1-8-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
jic062-nemo-v1-8-v1-mkmlizer: ║                                                                     ║
jic062-nemo-v1-8-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
jic062-nemo-v1-8-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-nemo-v1-8-v2-mkmlizer: ║     _____            __           __                                ║
jic062-nemo-v1-8-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
jic062-nemo-v1-8-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
jic062-nemo-v1-8-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
jic062-nemo-v1-8-v2-mkmlizer: ║       /___/                                                         ║
jic062-nemo-v1-8-v2-mkmlizer: ║                                                                     ║
jic062-nemo-v1-8-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
jic062-nemo-v1-8-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
jic062-nemo-v1-8-v2-mkmlizer: ║  https://mk1.ai                                                     ║
jic062-nemo-v1-8-v2-mkmlizer: ║                                                                     ║
jic062-nemo-v1-8-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
jic062-nemo-v1-8-v2-mkmlizer: ║  belonging to:                                                      ║
jic062-nemo-v1-8-v2-mkmlizer: ║                                                                     ║
jic062-nemo-v1-8-v2-mkmlizer: ║  Chai Research Corp.                                                ║
jic062-nemo-v1-8-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
jic062-nemo-v1-8-v2-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
jic062-nemo-v1-8-v2-mkmlizer: ║                                                                     ║
jic062-nemo-v1-8-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
jic062-nemo-v1-8-v1-mkmlizer: Downloaded to shared memory in 56.379s
jic062-nemo-v1-8-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmps9jpjxi5, device:0
jic062-nemo-v1-8-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
jic062-nemo-v1-8-v2-mkmlizer: Downloaded to shared memory in 45.063s
jic062-nemo-v1-8-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpwb0rgvsw, device:0
jic062-nemo-v1-8-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
jic062-nemo-v1-8-v2-mkmlizer: quantized model in 38.316s
jic062-nemo-v1-8-v2-mkmlizer: Processed model jic062/Nemo-v1.8 in 83.380s
jic062-nemo-v1-8-v2-mkmlizer: creating bucket guanaco-mkml-models
jic062-nemo-v1-8-v1-mkmlizer: quantized model in 37.967s
jic062-nemo-v1-8-v1-mkmlizer: Processed model jic062/Nemo-v1.8 in 94.346s
jic062-nemo-v1-8-v1-mkmlizer: creating bucket guanaco-mkml-models
jic062-nemo-v1-8-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-nemo-v1-8-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-nemo-v1-8-v1
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/config.json
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/special_tokens_map.json
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/tokenizer_config.json
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/tokenizer.json
jic062-nemo-v1-8-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
jic062-nemo-v1-8-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/jic062-nemo-v1-8-v2
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/config.json
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/special_tokens_map.json
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/tokenizer_config.json
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/tokenizer.json
jic062-nemo-v1-8-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/jic062-nemo-v1-8-v2/flywheel_model.0.safetensors
jic062-nemo-v1-8-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 27.99it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:07, 45.58it/s]
Loading 0:   5%|▍         | 18/363 [00:00<00:07, 44.70it/s]
Loading 0:   6%|▋         | 23/363 [00:00<00:09, 34.05it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 42.58it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:07, 41.30it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:07, 41.40it/s]
Loading 0:  12%|█▏        | 45/363 [00:01<00:07, 41.62it/s]
Loading 0:  14%|█▍        | 50/363 [00:01<00:09, 32.66it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:08, 37.13it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:10, 27.50it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:10, 27.31it/s]
Loading 0:  20%|█▉        | 71/363 [00:02<00:08, 32.88it/s]
Loading 0:  21%|██        | 75/363 [00:02<00:08, 32.66it/s]
Loading 0:  22%|██▏       | 80/363 [00:02<00:08, 34.76it/s]
Loading 0:  23%|██▎       | 84/363 [00:02<00:08, 33.63it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:07, 36.21it/s]
Loading 0:  26%|██▌       | 93/363 [00:02<00:07, 34.10it/s]
Loading 0:  27%|██▋       | 98/363 [00:02<00:07, 36.74it/s]
Loading 0:  28%|██▊       | 102/363 [00:02<00:07, 35.05it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:07, 35.67it/s]
Loading 0:  31%|███       | 112/363 [00:03<00:06, 39.88it/s]
Loading 0:  32%|███▏      | 117/363 [00:03<00:06, 37.95it/s]
Loading 0:  33%|███▎      | 121/363 [00:03<00:06, 37.09it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:06, 36.95it/s]
Loading 0:  36%|███▌      | 129/363 [00:03<00:06, 35.06it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:06, 37.34it/s]
Loading 0:  38%|███▊      | 138/363 [00:03<00:06, 34.63it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:08, 24.69it/s]
Loading 0:  40%|███▉      | 145/363 [00:04<00:08, 24.62it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:08, 25.01it/s]
Loading 0:  43%|████▎     | 156/363 [00:04<00:06, 33.03it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 32.18it/s]
Loading 0:  45%|████▌     | 165/363 [00:04<00:05, 33.18it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 32.55it/s]
Loading 0:  48%|████▊     | 174/363 [00:05<00:05, 35.42it/s]
Loading 0:  49%|████▉     | 178/363 [00:05<00:05, 33.69it/s]
Loading 0:  50%|█████     | 183/363 [00:05<00:05, 35.66it/s]
Loading 0:  52%|█████▏    | 187/363 [00:05<00:05, 34.39it/s]
Loading 0:  53%|█████▎    | 192/363 [00:05<00:04, 37.09it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:04, 35.09it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:04, 36.44it/s]
Loading 0:  56%|█████▋    | 205/363 [00:05<00:04, 34.36it/s]
Loading 0:  58%|█████▊    | 210/363 [00:06<00:04, 37.33it/s]
Loading 0:  59%|█████▉    | 214/363 [00:06<00:04, 35.13it/s]
Loading 0:  60%|██████    | 218/363 [00:06<00:04, 33.85it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:05, 26.17it/s]
Loading 0:  62%|██████▏   | 226/363 [00:06<00:05, 25.59it/s]
Loading 0:  63%|██████▎   | 230/363 [00:06<00:05, 25.91it/s]
Loading 0:  65%|██████▌   | 237/363 [00:07<00:03, 33.60it/s]
Loading 0:  66%|██████▋   | 241/363 [00:07<00:03, 32.83it/s]
Loading 0:  68%|██████▊   | 246/363 [00:07<00:03, 35.89it/s]
Loading 0:  69%|██████▉   | 250/363 [00:07<00:03, 34.71it/s]
Loading 0:  70%|███████   | 255/363 [00:07<00:02, 36.93it/s]
Loading 0:  71%|███████▏  | 259/363 [00:07<00:02, 35.17it/s]
Loading 0:  73%|███████▎  | 264/363 [00:07<00:02, 37.58it/s]
Loading 0:  74%|███████▍  | 268/363 [00:07<00:02, 35.89it/s]
Loading 0:  75%|███████▌  | 273/363 [00:07<00:02, 37.96it/s]
Loading 0:  76%|███████▋  | 277/363 [00:08<00:02, 35.36it/s]
Loading 0:  78%|███████▊  | 282/363 [00:08<00:02, 37.71it/s]
Loading 0:  79%|███████▉  | 286/363 [00:08<00:02, 36.01it/s]
Loading 0:  80%|████████  | 291/363 [00:08<00:01, 38.39it/s]
Loading 0:  81%|████████▏ | 295/363 [00:08<00:01, 36.43it/s]
Loading 0:  82%|████████▏ | 299/363 [00:08<00:01, 36.29it/s]
Loading 0:  84%|████████▎ | 304/363 [00:15<00:27,  2.11it/s]
Loading 0:  85%|████████▍ | 307/363 [00:15<00:21,  2.64it/s]
Loading 0:  86%|████████▌ | 312/363 [00:15<00:13,  3.88it/s]
Loading 0:  88%|████████▊ | 319/363 [00:15<00:06,  6.31it/s]
Loading 0:  89%|████████▉ | 323/363 [00:16<00:05,  7.92it/s]
Loading 0:  90%|█████████ | 328/363 [00:16<00:03, 10.69it/s]
Loading 0:  92%|█████████▏| 333/363 [00:16<00:02, 13.76it/s]
Loading 0:  93%|█████████▎| 338/363 [00:16<00:01, 17.03it/s]
Loading 0:  94%|█████████▍| 342/363 [00:16<00:01, 19.75it/s]
Loading 0:  95%|█████████▌| 346/363 [00:16<00:00, 22.84it/s]
Loading 0:  96%|█████████▋| 350/363 [00:16<00:00, 24.84it/s]
Loading 0:  98%|█████████▊| 355/363 [00:16<00:00, 29.65it/s]
Loading 0:  99%|█████████▉| 360/363 [00:16<00:00, 31.95it/s]
                                                            
jic062-nemo-v1-8-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/jic062-nemo-v1-8-v1/flywheel_model.0.safetensors
jic062-nemo-v1-8-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 28.32it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 46.68it/s]
Loading 0:   5%|▍         | 18/363 [00:00<00:07, 45.92it/s]
Loading 0:   6%|▋         | 23/363 [00:00<00:09, 34.74it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 43.31it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:07, 41.76it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:07, 41.60it/s]
Loading 0:  12%|█▏        | 45/363 [00:01<00:07, 41.90it/s]
Loading 0:  14%|█▍        | 50/363 [00:01<00:09, 32.68it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:08, 37.21it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:11, 25.88it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:11, 26.32it/s]
Loading 0:  20%|█▉        | 71/363 [00:02<00:09, 32.21it/s]
Loading 0:  21%|██        | 75/363 [00:02<00:09, 31.56it/s]
Loading 0:  22%|██▏       | 80/363 [00:02<00:08, 35.28it/s]
Loading 0:  23%|██▎       | 84/363 [00:02<00:08, 34.46it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:07, 37.54it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:07, 37.40it/s]
Loading 0:  27%|██▋       | 98/363 [00:02<00:07, 37.18it/s]
Loading 0:  28%|██▊       | 102/363 [00:02<00:07, 35.52it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:07, 36.32it/s]
Loading 0:  31%|███       | 112/363 [00:03<00:06, 39.85it/s]
Loading 0:  32%|███▏      | 117/363 [00:03<00:06, 37.34it/s]
Loading 0:  33%|███▎      | 121/363 [00:03<00:06, 37.22it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:06, 37.55it/s]
Loading 0:  36%|███▌      | 129/363 [00:03<00:06, 35.70it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:06, 38.01it/s]
Loading 0:  38%|███▊      | 138/363 [00:03<00:06, 35.17it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:09, 23.79it/s]
Loading 0:  40%|███▉      | 145/363 [00:04<00:09, 24.14it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:08, 24.92it/s]
Loading 0:  43%|████▎     | 156/363 [00:04<00:06, 33.38it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 31.60it/s]
Loading 0:  45%|████▌     | 165/363 [00:04<00:05, 33.82it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 33.18it/s]
Loading 0:  48%|████▊     | 174/363 [00:05<00:05, 35.97it/s]
Loading 0:  49%|████▉     | 178/363 [00:05<00:05, 34.30it/s]
Loading 0:  50%|█████     | 183/363 [00:05<00:04, 36.08it/s]
Loading 0:  52%|█████▏    | 187/363 [00:05<00:05, 34.77it/s]
Loading 0:  53%|█████▎    | 192/363 [00:05<00:04, 37.38it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:04, 35.12it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:04, 36.28it/s]
Loading 0:  56%|█████▋    | 205/363 [00:05<00:04, 34.84it/s]
Loading 0:  58%|█████▊    | 210/363 [00:06<00:04, 37.83it/s]
Loading 0:  59%|█████▉    | 214/363 [00:06<00:04, 36.20it/s]
Loading 0:  60%|██████    | 218/363 [00:06<00:04, 35.73it/s]
Loading 0:  61%|██████    | 222/363 [00:06<00:03, 36.45it/s]
Loading 0:  62%|██████▏   | 226/363 [00:06<00:05, 23.38it/s]
Loading 0:  63%|██████▎   | 230/363 [00:06<00:05, 24.25it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 32.13it/s]
Loading 0:  66%|██████▋   | 241/363 [00:07<00:03, 31.99it/s]
Loading 0:  68%|██████▊   | 246/363 [00:07<00:03, 35.24it/s]
Loading 0:  69%|██████▉   | 250/363 [00:07<00:03, 34.18it/s]
Loading 0:  70%|███████   | 255/363 [00:07<00:02, 37.07it/s]
Loading 0:  71%|███████▏  | 259/363 [00:07<00:02, 35.12it/s]
Loading 0:  73%|███████▎  | 264/363 [00:07<00:02, 37.33it/s]
Loading 0:  74%|███████▍  | 268/363 [00:07<00:02, 35.58it/s]
Loading 0:  75%|███████▌  | 273/363 [00:07<00:02, 36.49it/s]
Loading 0:  76%|███████▋  | 277/363 [00:08<00:02, 35.18it/s]
Loading 0:  78%|███████▊  | 282/363 [00:08<00:02, 38.35it/s]
Loading 0:  79%|███████▉  | 286/363 [00:08<00:02, 36.74it/s]
Loading 0:  80%|████████  | 291/363 [00:08<00:01, 38.73it/s]
Loading 0:  81%|████████▏ | 295/363 [00:08<00:01, 36.54it/s]
Loading 0:  82%|████████▏ | 299/363 [00:08<00:01, 36.54it/s]
Loading 0:  84%|████████▎ | 304/363 [00:15<00:28,  2.10it/s]
Loading 0:  85%|████████▍ | 307/363 [00:15<00:21,  2.64it/s]
Loading 0:  86%|████████▌ | 312/363 [00:15<00:13,  3.88it/s]
Loading 0:  88%|████████▊ | 319/363 [00:15<00:06,  6.32it/s]
Loading 0:  89%|████████▉ | 324/363 [00:16<00:04,  8.40it/s]
Loading 0:  91%|█████████ | 329/363 [00:16<00:03, 11.03it/s]
Loading 0:  92%|█████████▏| 334/363 [00:16<00:02, 14.22it/s]
Loading 0:  93%|█████████▎| 339/363 [00:16<00:01, 16.08it/s]
Loading 0:  95%|█████████▌| 346/363 [00:16<00:00, 22.35it/s]
Loading 0:  97%|█████████▋| 351/363 [00:16<00:00, 25.40it/s]
Loading 0:  98%|█████████▊| 356/363 [00:16<00:00, 28.74it/s]
Loading 0:  99%|█████████▉| 361/363 [00:16<00:00, 31.29it/s]
                                                            
Job jic062-nemo-v1-8-v1-mkmlizer completed after 124.64s with status: succeeded
Stopping job with name jic062-nemo-v1-8-v1-mkmlizer
Pipeline stage MKMLizer completed in 125.51s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service jic062-nemo-v1-8-v1
Waiting for inference service jic062-nemo-v1-8-v1 to be ready
Job jic062-nemo-v1-8-v2-mkmlizer completed after 113.91s with status: succeeded
Stopping job with name jic062-nemo-v1-8-v2-mkmlizer
Pipeline stage MKMLizer completed in 114.30s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service jic062-nemo-v1-8-v2
Waiting for inference service jic062-nemo-v1-8-v2 to be ready
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name jic062-nemo-v1-6-v4-mkmlizer
Waiting for job on jic062-nemo-v1-6-v4-mkmlizer to finish
jic062-nemo-v1-6-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
jic062-nemo-v1-6-v4-mkmlizer: ║     _____            __           __                                ║
jic062-nemo-v1-6-v4-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
jic062-nemo-v1-6-v4-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
jic062-nemo-v1-6-v4-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
jic062-nemo-v1-6-v4-mkmlizer: ║       /___/                                                         ║
jic062-nemo-v1-6-v4-mkmlizer: ║                                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Version: 0.11.12                                                   ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
jic062-nemo-v1-6-v4-mkmlizer: ║  https://mk1.ai                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ║                                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ║  The license key for the current software has been verified as      ║
jic062-nemo-v1-6-v4-mkmlizer: ║  belonging to:                                                      ║
jic062-nemo-v1-6-v4-mkmlizer: ║                                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Chai Research Corp.                                                ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
jic062-nemo-v1-6-v4-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
jic062-nemo-v1-6-v4-mkmlizer: ║                                                                     ║
jic062-nemo-v1-6-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Inference service jic062-nemo-v1-8-v1 ready after 230.64079999923706s
Pipeline stage MKMLDeployer completed in 231.16s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3941752910614014s
Received healthy response to inference request in 1.8138670921325684s
Inference service jic062-nemo-v1-8-v2 ready after 230.6058521270752s
Received healthy response to inference request in 2.036547899246216s
Pipeline stage MKMLDeployer completed in 231.05s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.058387279510498s
Received healthy response to inference request in 2.402942180633545s
Received healthy response to inference request in 1.9270868301391602s
5 requests
0 failed requests
5th percentile: 1.8365110397338866
10th percentile: 1.8591549873352051
Received healthy response to inference request in 1.7296473979949951s
20th percentile: 1.9044428825378419
30th percentile: 1.9489790439605712
40th percentile: 1.9927634716033935
50th percentile: 2.036547899246216
60th percentile: 2.045283651351929
70th percentile: 2.054019403457642
80th percentile: 2.125544881820679
90th percentile: 2.25986008644104
95th percentile: 2.3270176887512206
99th percentile: 2.380743770599365
mean time: 2.0460128784179688
Pipeline stage StressChecker completed in 12.39s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
jic062-nemo-v1-6-v4-mkmlizer: Downloaded to shared memory in 62.361s
jic062-nemo-v1-6-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpuqdz21op, device:0
jic062-nemo-v1-6-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Received healthy response to inference request in 1.7409923076629639s
Received healthy response to inference request in 1.69749116897583s
Received healthy response to inference request in 1.7477772235870361s
5 requests
0 failed requests
5th percentile: 1.703922414779663
10th percentile: 1.7103536605834961
20th percentile: 1.7232161521911622
30th percentile: 1.7319163799285888
40th percentile: 1.7364543437957765
50th percentile: 1.7409923076629639
60th percentile: 1.7437062740325928
70th percentile: 1.7464202404022218
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.44s
80th percentile: 1.878810214996338
Shutdown handler de-registered
90th percentile: 2.1408761978149413
jic062-nemo-v1-8_v1 status is now deployed due to DeploymentManager action
95th percentile: 2.271909189224243
99th percentile: 2.3767355823516847
mean time: 1.863770055770874
Pipeline stage StressChecker completed in 11.74s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.97s
Shutdown handler de-registered
jic062-nemo-v1-8_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service jic062-nemo-v1-8-v2-profiler
Waiting for inference service jic062-nemo-v1-8-v2-profiler to be ready
Tearing down inference service jic062-nemo-v1-8-v2-profiler
%s, retrying in %s seconds...
Creating inference service jic062-nemo-v1-8-v2-profiler
Waiting for inference service jic062-nemo-v1-8-v2-profiler to be ready
Tearing down inference service jic062-nemo-v1-8-v2-profiler
%s, retrying in %s seconds...
Creating inference service jic062-nemo-v1-8-v2-profiler
Waiting for inference service jic062-nemo-v1-8-v2-profiler to be ready
Tearing down inference service jic062-nemo-v1-8-v2-profiler
clean up pipeline due to error=%s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
Shutdown handler de-registered
jic062-nemo-v1-8_v2 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a8769ce1-cf82-4e6c-a010-0d7f584eb54c, cf0aa001-7357-40f9-9001-24796db352c4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3a203de4-224d-49bc-872e-8867a5231874, 016a7cfe-5457-4989-9097-d50de61d1a81\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'48926ab1-06b9-4708-8551-0d28086e92e4, 87d12733-9360-4cd4-a87d-c45ffcac5740\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:53 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'04b132bb-2c48-4858-b6ed-da9522cdb2b2, a5f24bff-f9c8-413b-96a2-f28801dec4dc\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dae1e8bd-8fd7-45d1-b101-bf166f33a4be, 4a501989-f9ad-467d-9042-a4cd2a899c5b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:35:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'39080258-0a3e-4d6b-ad23-e5159d96c6d2, a96d1611-2ecb-44fe-afe8-bacee251670f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:40:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a561bda5-7402-45e7-86aa-a2cea92421b8, 138a2718-625f-42ff-a524-2bb2518ee40d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:45:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'37852995-fd4b-46d3-a65d-fbf57b44a74c, bfbf881f-8551-4053-b244-719fb74577bb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:50:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'07b157e6-be8b-4c1b-bd41-70be1ef665a0, bc632ef2-6704-4d91-b5a6-59cede92cec0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:55:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0357e71f-6b62-4de1-b663-3ce2dd19e4ee, 88641aee-fee0-4814-8e8a-f77a8ed94714\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2e4a001-b46d-488f-b2ac-5934dcc38bfb, 9784b871-0e9b-48d0-bcd7-5de7e30a8696\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'443014ae-6253-4c13-ba74-92cd8b560bba, c657ed4d-a233-4d38-93e7-feaf5ad40808\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:02 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'245e01bd-3aa4-40bf-bebc-ea0bf6fd2d98, b1fc41b2-1a18-49c9-90ca-e99d941a25e0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e99e695a-b8c9-4ffb-8988-f066a235ad07, bb73b31e-de58-4c25-a477-ee8f12075588\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'830389ff-8ee5-4ed2-9e40-722d0151f050, 24bd5d9b-45e5-44a7-a730-62e2e17ef4d1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'997bb99b-2e2a-4d86-94c3-0bcdcc373a75, a13afc69-2069-4d54-b7c2-8aeb512ac36f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'70d7cfec-dd23-42e7-85d6-3a680543239e, 8380e93d-abca-4e05-918d-0bbd2cd89930\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0f6b26fc-97be-49d0-a24f-199d27a1a7d0, 6b3c5fbd-4e5f-436e-a3bf-36ea3c4cc0cf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:11 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e459d4c1-2e60-44b9-8d58-66aa782a8fb7, c41c0a33-5821-4f4a-a906-2b68a72f9098\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4c67dd38-6497-4adf-a9a3-73350f19f2a5, d1d063f8-f14f-4c69-be4e-7f9169910947\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1f999239-a4d3-4072-b976-352846031a0d, 2927f086-f9c2-4444-adce-c1f33b994a43\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:12 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0ae2237b-6339-45a0-9646-a7e9ac685e66, c11dd34e-8070-4fed-a1c7-ca6542d88414\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1bedf0c4-16d1-43f9-892a-4243c629b0a8, 9a1da1a6-6200-4eb1-a2a2-a6f2a86d3b97\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'055eb5f3-8de9-4c7e-bbae-f78d99880a69, 780fa44d-a36b-4a7a-9584-41ce61e39b9c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c77cdf2e-d5b9-44f9-9e7c-60a6bacf4f8c, 74ceca4c-5b15-4b3b-a205-e85f0131f23f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
admin requested tearing down of jic062-nemo-v1-9_v6
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage MKMLDeleter
run pipeline %s
Checking if service jic062-nemo-v1-8-v2 is running
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
Checking if service jic062-nemo-v1-9-v6 is running
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
Checking if service jic062-nemo-v1-9-v6 is running
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'da6d1ff5-30c3-446a-be68-bc0167368059, 9a0ec40f-e1c0-4f6e-a40e-b1f47483d1dc\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Checking if service jic062-nemo-v1-9-v6 is running
Shutdown handler de-registered
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bbbb775e-e77d-424a-8357-3b74d88278f9, aed56b0e-88ee-4139-a71a-007ddfea2f9a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
admin requested tearing down of jic062-nemo-v1-8_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
%s, retrying in %s seconds...
Checking if service jic062-nemo-v1-8-v2 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c8c0fd28-8faa-461b-95bf-31c6be95b7c8, 10701545-c2c3-4d62-b47b-9dc815be5fec\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
jic062-nemo-v1-8_v2 status is now torndown due to DeploymentManager action