chaiml-nemo-community-2c

developer_uid: azuruce
submission_id: chaiml-nemo-community-2c_v1
model_name: starcannon-mix
model_group: ChaiML/nemo-community-2c
status: torndown
timestamp: 2024-09-30T22:41:47+00:00
num_battles: 3898
num_wins: 2004
celo_rating: 1265.63
family_friendly_score: 0.5524552587498898
family_friendly_standard_error: 0.008066086018445025
submission_type: basic
model_repo: ChaiML/nemo-community-2c
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: starcannon-mix
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: ChaiML/nemo-community-2c
model_size: 13B
ranking_group: single
us_pacific_date: 2024-09-30
win_ratio: 0.5141097998973833
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>', '<|end_of_text|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-nemo-community-2c-v1-mkmlizer
Waiting for job on chaiml-nemo-community-2c-v1-mkmlizer to finish
chaiml-nemo-community-2c-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-nemo-community-2c-v1-mkmlizer: ║     _____            __           __                                ║
chaiml-nemo-community-2c-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-nemo-community-2c-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-nemo-community-2c-v1-mkmlizer: ║       /___/                                                         ║
chaiml-nemo-community-2c-v1-mkmlizer: ║                                                                     ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-nemo-community-2c-v1-mkmlizer: ║                                                                     ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-nemo-community-2c-v1-mkmlizer: ║                                                                     ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-nemo-community-2c-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-nemo-community-2c-v1-mkmlizer: ║                                                                     ║
chaiml-nemo-community-2c-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-nemo-community-2a-v1-mkmlizer: Downloaded to shared memory in 48.848s
chaiml-nemo-community-2a-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpyol4dkqd, device:0
chaiml-nemo-community-2a-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nemo-community-2b-v1-mkmlizer: Downloaded to shared memory in 49.498s
chaiml-nemo-community-2b-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp0kzmfyfc, device:0
chaiml-nemo-community-2b-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nemo-community-2a-v1-mkmlizer: quantized model in 35.722s
chaiml-nemo-community-2a-v1-mkmlizer: Processed model ChaiML/nemo-community-2a in 84.570s
chaiml-nemo-community-2a-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nemo-community-2a-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nemo-community-2a-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nemo-community-2a-v1
chaiml-nemo-community-2a-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nemo-community-2a-v1/config.json
chaiml-nemo-community-2a-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nemo-community-2a-v1/special_tokens_map.json
chaiml-nemo-community-2a-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nemo-community-2a-v1/tokenizer_config.json
chaiml-nemo-community-2a-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-nemo-community-2a-v1/tokenizer.json
chaiml-nemo-community-2a-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nemo-community-2a-v1/flywheel_model.0.safetensors
chaiml-nemo-community-2a-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:05<17:50,  2.96s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:46,  1.25it/s]
Loading 0:   3%|▎         | 11/363 [00:06<02:05,  2.80it/s]
Loading 0:   4%|▍         | 15/363 [00:06<01:19,  4.36it/s]
Loading 0:   6%|▋         | 23/363 [00:06<00:39,  8.66it/s]
Loading 0:   8%|▊         | 28/363 [00:06<00:28, 11.75it/s]
Loading 0:   9%|▉         | 33/363 [00:06<00:23, 14.31it/s]
Loading 0:  11%|█         | 40/363 [00:06<00:18, 17.69it/s]
Loading 0:  12%|█▏        | 44/363 [00:07<00:15, 20.09it/s]
Loading 0:  14%|█▍        | 50/363 [00:07<00:12, 25.43it/s]
Loading 0:  15%|█▌        | 56/363 [00:07<00:10, 29.18it/s]
Loading 0:  17%|█▋        | 61/363 [00:07<00:09, 30.69it/s]
Loading 0:  18%|█▊        | 67/363 [00:07<00:08, 36.36it/s]
Loading 0:  20%|█▉        | 72/363 [00:07<00:07, 39.06it/s]
Loading 0:  21%|██        | 77/363 [00:07<00:06, 41.35it/s]
Loading 0:  23%|██▎       | 83/363 [00:07<00:06, 41.51it/s]
Loading 0:  24%|██▍       | 88/363 [00:08<00:06, 41.94it/s]
Loading 0:  26%|██▌       | 94/363 [00:08<00:05, 46.21it/s]
Loading 0:  27%|██▋       | 99/363 [00:08<00:05, 46.45it/s]
Loading 0:  29%|██▊       | 104/363 [00:08<00:05, 47.22it/s]
Loading 0:  30%|███       | 110/363 [00:08<00:05, 45.29it/s]
Loading 0:  32%|███▏      | 115/363 [00:08<00:05, 42.33it/s]
Loading 0:  33%|███▎      | 121/363 [00:08<00:07, 32.30it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:07, 32.92it/s]
Loading 0:  36%|███▌      | 131/363 [00:09<00:06, 37.69it/s]
Loading 0:  38%|███▊      | 137/363 [00:09<00:05, 38.87it/s]
Loading 0:  39%|███▉      | 142/363 [00:09<00:05, 39.81it/s]
Loading 0:  41%|████      | 149/363 [00:09<00:04, 45.93it/s]
Loading 0:  43%|████▎     | 155/363 [00:09<00:04, 43.42it/s]
Loading 0:  44%|████▍     | 160/363 [00:09<00:04, 41.67it/s]
Loading 0:  46%|████▌     | 167/363 [00:09<00:04, 47.05it/s]
Loading 0:  48%|████▊     | 173/363 [00:10<00:04, 45.61it/s]
Loading 0:  49%|████▉     | 178/363 [00:10<00:04, 43.07it/s]
Loading 0:  51%|█████     | 185/363 [00:10<00:03, 48.05it/s]
Loading 0:  53%|█████▎    | 191/363 [00:10<00:03, 46.65it/s]
Loading 0:  54%|█████▍    | 196/363 [00:10<00:03, 45.60it/s]
Loading 0:  56%|█████▌    | 202/363 [00:10<00:04, 36.37it/s]
Loading 0:  57%|█████▋    | 206/363 [00:10<00:04, 35.59it/s]
Loading 0:  58%|█████▊    | 212/363 [00:11<00:03, 39.54it/s]
Loading 0:  60%|█████▉    | 217/363 [00:11<00:03, 41.92it/s]
Loading 0:  61%|██████    | 222/363 [00:11<00:03, 35.62it/s]
Loading 0:  63%|██████▎   | 229/363 [00:11<00:03, 42.12it/s]
Loading 0:  64%|██████▍   | 234/363 [00:11<00:03, 42.52it/s]
Loading 0:  66%|██████▌   | 239/363 [00:11<00:02, 43.70it/s]
Loading 0:  67%|██████▋   | 244/363 [00:11<00:02, 45.18it/s]
Loading 0:  69%|██████▊   | 249/363 [00:11<00:02, 38.01it/s]
Loading 0:  71%|███████   | 257/363 [00:12<00:02, 46.46it/s]
Loading 0:  72%|███████▏  | 262/363 [00:12<00:02, 46.25it/s]
Loading 0:  74%|███████▎  | 267/363 [00:12<00:02, 39.07it/s]
Loading 0:  75%|███████▌  | 274/363 [00:12<00:01, 45.78it/s]
Loading 0:  77%|███████▋  | 279/363 [00:12<00:01, 43.94it/s]
Loading 0:  78%|███████▊  | 284/363 [00:12<00:02, 31.87it/s]
Loading 0:  79%|███████▉  | 288/363 [00:12<00:02, 33.25it/s]
Loading 0:  81%|████████  | 293/363 [00:13<00:01, 36.20it/s]
Loading 0:  82%|████████▏ | 298/363 [00:13<00:01, 38.66it/s]
Loading 0:  83%|████████▎ | 303/363 [00:13<00:01, 33.96it/s]
Loading 0:  85%|████████▌ | 310/363 [00:13<00:01, 41.80it/s]
Loading 0:  87%|████████▋ | 315/363 [00:13<00:01, 41.73it/s]
Loading 0:  88%|████████▊ | 320/363 [00:13<00:00, 43.06it/s]
Loading 0:  90%|████████▉ | 325/363 [00:13<00:00, 44.06it/s]
Loading 0:  91%|█████████ | 330/363 [00:14<00:00, 36.92it/s]
Loading 0:  93%|█████████▎| 337/363 [00:14<00:00, 44.37it/s]
Loading 0:  94%|█████████▍| 342/363 [00:14<00:00, 43.12it/s]
Loading 0:  96%|█████████▌| 347/363 [00:14<00:00, 42.96it/s]
Loading 0:  97%|█████████▋| 352/363 [00:14<00:00, 43.76it/s]
Loading 0:  98%|█████████▊| 357/363 [00:14<00:00, 37.89it/s]
                                                            
Job chaiml-nemo-community-2a-v1-mkmlizer completed after 113.62s with status: succeeded
Stopping job with name chaiml-nemo-community-2a-v1-mkmlizer
Pipeline stage MKMLizer completed in 114.73s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nemo-community-2a-v1
Waiting for inference service chaiml-nemo-community-2a-v1 to be ready
chaiml-nemo-community-2c-v1-mkmlizer: Downloaded to shared memory in 55.158s
chaiml-nemo-community-2c-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpahmv7r30, device:0
chaiml-nemo-community-2c-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nemo-community-2b-v1-mkmlizer: quantized model in 36.445s
chaiml-nemo-community-2b-v1-mkmlizer: Processed model ChaiML/nemo-community-2b in 85.943s
chaiml-nemo-community-2b-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nemo-community-2b-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nemo-community-2b-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nemo-community-2b-v1
chaiml-nemo-community-2b-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nemo-community-2b-v1/config.json
chaiml-nemo-community-2b-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nemo-community-2b-v1/special_tokens_map.json
chaiml-nemo-community-2b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nemo-community-2b-v1/tokenizer_config.json
chaiml-nemo-community-2b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-nemo-community-2b-v1/tokenizer.json
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name marinaraspaghetti-nemom-1739-v10-mkmlizer
Waiting for job on marinaraspaghetti-nemom-1739-v10-mkmlizer to finish
chaiml-nemo-community-2b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nemo-community-2b-v1/flywheel_model.0.safetensors
chaiml-nemo-community-2b-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:04,  3.00s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:49,  1.23it/s]
Loading 0:   4%|▎         | 13/363 [00:06<01:43,  3.39it/s]
Loading 0:   5%|▍         | 17/363 [00:06<01:09,  4.96it/s]
Loading 0:   6%|▋         | 23/363 [00:06<00:42,  8.08it/s]
Loading 0:   8%|▊         | 28/363 [00:06<00:29, 11.17it/s]
Loading 0:   9%|▉         | 33/363 [00:06<00:24, 13.67it/s]
Loading 0:  11%|█         | 40/363 [00:07<00:19, 16.79it/s]
Loading 0:  12%|█▏        | 44/363 [00:07<00:16, 19.27it/s]
Loading 0:  14%|█▍        | 50/363 [00:07<00:12, 24.55it/s]
Loading 0:  15%|█▌        | 55/363 [00:07<00:10, 28.73it/s]
Loading 0:  17%|█▋        | 60/363 [00:07<00:10, 27.96it/s]
Loading 0:  18%|█▊        | 67/363 [00:07<00:08, 34.76it/s]
Loading 0:  20%|█▉        | 72/363 [00:07<00:07, 36.84it/s]
Loading 0:  21%|██        | 77/363 [00:07<00:07, 38.24it/s]
Loading 0:  23%|██▎       | 83/363 [00:08<00:07, 38.20it/s]
Loading 0:  24%|██▍       | 88/363 [00:08<00:07, 37.56it/s]
Loading 0:  26%|██▌       | 94/363 [00:08<00:06, 42.45it/s]
Loading 0:  27%|██▋       | 99/363 [00:08<00:06, 43.38it/s]
Loading 0:  29%|██▊       | 104/363 [00:08<00:05, 44.70it/s]
Loading 0:  30%|███       | 109/363 [00:08<00:05, 45.61it/s]
Loading 0:  31%|███▏      | 114/363 [00:08<00:07, 35.55it/s]
Loading 0:  33%|███▎      | 121/363 [00:09<00:07, 30.85it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:07, 31.67it/s]
Loading 0:  36%|███▌      | 131/363 [00:09<00:06, 36.14it/s]
Loading 0:  37%|███▋      | 136/363 [00:09<00:05, 38.98it/s]
Loading 0:  39%|███▉      | 141/363 [00:09<00:06, 32.89it/s]
Loading 0:  41%|████      | 148/363 [00:09<00:05, 39.59it/s]
Loading 0:  42%|████▏     | 153/363 [00:09<00:05, 40.08it/s]
Loading 0:  44%|████▎     | 158/363 [00:10<00:05, 40.48it/s]
Loading 0:  45%|████▍     | 163/363 [00:10<00:04, 42.32it/s]
Loading 0:  46%|████▋     | 168/363 [00:10<00:05, 35.97it/s]
Loading 0:  48%|████▊     | 175/363 [00:10<00:04, 43.12it/s]
Loading 0:  50%|████▉     | 180/363 [00:10<00:04, 43.20it/s]
Loading 0:  51%|█████     | 185/363 [00:10<00:04, 42.98it/s]
Loading 0:  52%|█████▏    | 190/363 [00:10<00:03, 44.08it/s]
Loading 0:  54%|█████▎    | 195/363 [00:10<00:04, 37.15it/s]
Loading 0:  56%|█████▌    | 202/363 [00:11<00:05, 31.50it/s]
Loading 0:  57%|█████▋    | 206/363 [00:11<00:04, 32.21it/s]
Loading 0:  58%|█████▊    | 211/363 [00:11<00:04, 35.90it/s]
Loading 0:  59%|█████▉    | 215/363 [00:11<00:04, 36.06it/s]
Loading 0:  61%|██████    | 220/363 [00:11<00:03, 38.41it/s]
Loading 0:  62%|██████▏   | 225/363 [00:11<00:03, 38.53it/s]
Loading 0:  63%|██████▎   | 230/363 [00:11<00:03, 39.75it/s]
Loading 0:  65%|██████▍   | 235/363 [00:12<00:03, 41.76it/s]
Loading 0:  66%|██████▌   | 240/363 [00:12<00:03, 33.81it/s]
Loading 0:  68%|██████▊   | 247/363 [00:12<00:02, 40.38it/s]
Loading 0:  69%|██████▉   | 252/363 [00:12<00:02, 40.11it/s]
Loading 0:  71%|███████   | 257/363 [00:12<00:02, 40.26it/s]
Loading 0:  72%|███████▏  | 262/363 [00:12<00:02, 41.55it/s]
Loading 0:  74%|███████▎  | 267/363 [00:12<00:02, 34.60it/s]
Loading 0:  75%|███████▌  | 274/363 [00:13<00:02, 41.37it/s]
Loading 0:  77%|███████▋  | 279/363 [00:13<00:02, 40.72it/s]
Loading 0:  78%|███████▊  | 284/363 [00:13<00:02, 29.48it/s]
Loading 0:  79%|███████▉  | 288/363 [00:13<00:02, 31.33it/s]
Loading 0:  80%|████████  | 292/363 [00:13<00:02, 32.98it/s]
Loading 0:  82%|████████▏ | 296/363 [00:13<00:02, 32.38it/s]
Loading 0:  83%|████████▎ | 301/363 [00:13<00:01, 35.60it/s]
Loading 0:  84%|████████▍ | 305/363 [00:14<00:01, 35.42it/s]
Loading 0:  85%|████████▌ | 310/363 [00:14<00:01, 38.30it/s]
Loading 0:  87%|████████▋ | 314/363 [00:14<00:01, 36.97it/s]
Loading 0:  88%|████████▊ | 319/363 [00:14<00:01, 39.08it/s]
Loading 0:  89%|████████▉ | 323/363 [00:14<00:01, 37.52it/s]
Loading 0:  90%|█████████ | 328/363 [00:14<00:00, 39.63it/s]
Loading 0:  92%|█████████▏| 333/363 [00:14<00:00, 38.75it/s]
Loading 0:  93%|█████████▎| 337/363 [00:14<00:00, 38.14it/s]
Loading 0:  94%|█████████▍| 341/363 [00:14<00:00, 36.31it/s]
Loading 0:  95%|█████████▌| 346/363 [00:15<00:00, 37.97it/s]
Loading 0:  96%|█████████▋| 350/363 [00:15<00:00, 36.48it/s]
Loading 0:  98%|█████████▊| 355/363 [00:15<00:00, 38.59it/s]
Loading 0:  99%|█████████▉| 359/363 [00:15<00:00, 37.44it/s]
                                                            
Job chaiml-nemo-community-2b-v1-mkmlizer completed after 114.33s with status: succeeded
Stopping job with name chaiml-nemo-community-2b-v1-mkmlizer
Pipeline stage MKMLizer completed in 115.10s
run pipeline stage %s
Running pipeline stage MKMLTemplater
marinaraspaghetti-nemom-1739-v10-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║     _____            __           __                                ║
Pipeline stage MKMLTemplater completed in 0.29s
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
run pipeline stage %s
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
Running pipeline stage MKMLDeployer
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
Creating inference service chaiml-nemo-community-2b-v1
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║       /___/                                                         ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║                                                                     ║
Waiting for inference service chaiml-nemo-community-2b-v1 to be ready
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  Version: 0.11.12                                                   ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  https://mk1.ai                                                     ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║                                                                     ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  The license key for the current software has been verified as      ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  belonging to:                                                      ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║                                                                     ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  Chai Research Corp.                                                ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ║                                                                     ║
marinaraspaghetti-nemom-1739-v10-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name intervitens-mini-magnum-5180-v7-mkmlizer
Waiting for job on intervitens-mini-magnum-5180-v7-mkmlizer to finish
chaiml-nemo-community-2c-v1-mkmlizer: quantized model in 36.417s
chaiml-nemo-community-2c-v1-mkmlizer: Processed model ChaiML/nemo-community-2c in 91.575s
chaiml-nemo-community-2c-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nemo-community-2c-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nemo-community-2c-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nemo-community-2c-v1
chaiml-nemo-community-2c-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nemo-community-2c-v1/special_tokens_map.json
chaiml-nemo-community-2c-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nemo-community-2c-v1/config.json
chaiml-nemo-community-2c-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nemo-community-2c-v1/tokenizer_config.json
chaiml-nemo-community-2c-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-nemo-community-2c-v1/tokenizer.json
intervitens-mini-magnum-5180-v7-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
intervitens-mini-magnum-5180-v7-mkmlizer: ║     _____            __           __                                ║
intervitens-mini-magnum-5180-v7-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
intervitens-mini-magnum-5180-v7-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
intervitens-mini-magnum-5180-v7-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
intervitens-mini-magnum-5180-v7-mkmlizer: ║       /___/                                                         ║
intervitens-mini-magnum-5180-v7-mkmlizer: ║                                                                     ║
intervitens-mini-magnum-5180-v7-mkmlizer: ║  Version: 0.11.12                                                   ║
intervitens-mini-magnum-5180-v7-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
intervitens-mini-magnum-5180-v7-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-nemo-community-2c-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nemo-community-2c-v1/flywheel_model.0.safetensors
intervitens-mini-magnum-5180-v7-mkmlizer: ║                                                                     ║
chaiml-nemo-community-2c-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:05<17:50,  2.97s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:46,  1.25it/s]
Loading 0:   3%|▎         | 11/363 [00:06<02:05,  2.80it/s]
Loading 0:   4%|▍         | 15/363 [00:06<01:20,  4.34it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:42,  8.09it/s]
Loading 0:   7%|▋         | 27/363 [00:06<00:30, 11.10it/s]
Loading 0:   9%|▉         | 32/363 [00:06<00:22, 14.61it/s]
Loading 0:  10%|█         | 37/363 [00:06<00:17, 18.76it/s]
Loading 0:  12%|█▏        | 42/363 [00:07<00:18, 16.91it/s]
Loading 0:  13%|█▎        | 49/363 [00:07<00:13, 23.72it/s]
Loading 0:  15%|█▍        | 54/363 [00:07<00:11, 25.85it/s]
Loading 0:  16%|█▋        | 59/363 [00:07<00:10, 28.91it/s]
Loading 0:  18%|█▊        | 64/363 [00:07<00:09, 32.81it/s]
Loading 0:  19%|█▉        | 69/363 [00:07<00:09, 30.08it/s]
Loading 0:  21%|██        | 76/363 [00:07<00:07, 36.36it/s]
Loading 0:  22%|██▏       | 81/363 [00:08<00:07, 37.93it/s]
Loading 0:  24%|██▎       | 86/363 [00:08<00:06, 39.81it/s]
Loading 0:  25%|██▌       | 91/363 [00:08<00:06, 41.96it/s]
Loading 0:  26%|██▋       | 96/363 [00:08<00:07, 35.31it/s]
Loading 0:  29%|██▊       | 104/363 [00:08<00:05, 43.47it/s]
Loading 0:  30%|███       | 109/363 [00:08<00:05, 44.96it/s]
Loading 0:  31%|███▏      | 114/363 [00:08<00:06, 37.20it/s]
Loading 0:  33%|███▎      | 119/363 [00:09<00:06, 39.52it/s]
Loading 0:  34%|███▍      | 124/363 [00:09<00:08, 28.96it/s]
Loading 0:  36%|███▌      | 130/363 [00:09<00:06, 34.30it/s]
Loading 0:  37%|███▋      | 135/363 [00:09<00:06, 36.33it/s]
Loading 0:  39%|███▊      | 140/363 [00:09<00:05, 37.71it/s]
Loading 0:  40%|███▉      | 145/363 [00:09<00:05, 40.57it/s]
Loading 0:  41%|████▏     | 150/363 [00:09<00:06, 35.49it/s]
Loading 0:  43%|████▎     | 157/363 [00:10<00:04, 41.75it/s]
Loading 0:  45%|████▍     | 162/363 [00:10<00:04, 42.34it/s]
Loading 0:  46%|████▌     | 167/363 [00:10<00:04, 43.01it/s]
Loading 0:  48%|████▊     | 173/363 [00:10<00:04, 40.97it/s]
Loading 0:  49%|████▉     | 178/363 [00:10<00:04, 39.14it/s]
Loading 0:  50%|█████     | 183/363 [00:10<00:04, 41.58it/s]
Loading 0:  52%|█████▏    | 188/363 [00:10<00:04, 39.89it/s]
Loading 0:  53%|█████▎    | 193/363 [00:10<00:04, 41.96it/s]
Loading 0:  55%|█████▍    | 198/363 [00:11<00:03, 42.60it/s]
Loading 0:  56%|█████▌    | 203/363 [00:11<00:05, 31.07it/s]
Loading 0:  57%|█████▋    | 208/363 [00:11<00:04, 34.74it/s]
Loading 0:  58%|█████▊    | 212/363 [00:11<00:04, 35.41it/s]
Loading 0:  60%|██████    | 218/363 [00:11<00:04, 35.48it/s]
Loading 0:  61%|██████    | 222/363 [00:11<00:04, 34.02it/s]
Loading 0:  63%|██████▎   | 229/363 [00:11<00:03, 40.80it/s]
Loading 0:  64%|██████▍   | 234/363 [00:12<00:03, 40.77it/s]
Loading 0:  66%|██████▌   | 239/363 [00:12<00:03, 41.27it/s]
Loading 0:  67%|██████▋   | 244/363 [00:12<00:02, 40.17it/s]
Loading 0:  69%|██████▊   | 249/363 [00:12<00:03, 34.32it/s]
Loading 0:  71%|███████   | 257/363 [00:12<00:02, 42.77it/s]
Loading 0:  72%|███████▏  | 262/363 [00:12<00:02, 44.36it/s]
Loading 0:  74%|███████▎  | 267/363 [00:12<00:02, 36.29it/s]
Loading 0:  75%|███████▌  | 274/363 [00:13<00:02, 43.22it/s]
Loading 0:  77%|███████▋  | 279/363 [00:13<00:01, 43.35it/s]
Loading 0:  78%|███████▊  | 284/363 [00:13<00:02, 31.28it/s]
Loading 0:  79%|███████▉  | 288/363 [00:13<00:02, 32.43it/s]
Loading 0:  81%|████████  | 293/363 [00:13<00:01, 35.43it/s]
Loading 0:  82%|████████▏ | 299/363 [00:13<00:01, 35.06it/s]
Loading 0:  83%|████████▎ | 303/363 [00:13<00:01, 33.50it/s]
Loading 0:  85%|████████▌ | 310/363 [00:14<00:01, 41.24it/s]
Loading 0:  87%|████████▋ | 315/363 [00:14<00:01, 42.24it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:00, 43.56it/s]
Loading 0:  90%|████████▉ | 325/363 [00:14<00:00, 44.74it/s]
Loading 0:  91%|█████████ | 330/363 [00:14<00:00, 36.34it/s]
Loading 0:  93%|█████████▎| 337/363 [00:14<00:00, 43.56it/s]
Loading 0:  94%|█████████▍| 342/363 [00:14<00:00, 43.17it/s]
Loading 0:  96%|█████████▌| 347/363 [00:14<00:00, 42.32it/s]
Loading 0:  97%|█████████▋| 352/363 [00:15<00:00, 43.05it/s]
Loading 0:  98%|█████████▊| 357/363 [00:15<00:00, 36.00it/s]
                                                            
intervitens-mini-magnum-5180-v7-mkmlizer: ║  The license key for the current software has been verified as      ║
Job chaiml-nemo-community-2c-v1-mkmlizer completed after 115.18s with status: succeeded
intervitens-mini-magnum-5180-v7-mkmlizer: ║  belonging to:                                                      ║
Stopping job with name chaiml-nemo-community-2c-v1-mkmlizer
intervitens-mini-magnum-5180-v7-mkmlizer: ║                                                                     ║
Pipeline stage MKMLizer completed in 115.94s
intervitens-mini-magnum-5180-v7-mkmlizer: ║  Chai Research Corp.                                                ║
run pipeline stage %s
intervitens-mini-magnum-5180-v7-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
Running pipeline stage MKMLTemplater
intervitens-mini-magnum-5180-v7-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
Pipeline stage MKMLTemplater completed in 0.24s
intervitens-mini-magnum-5180-v7-mkmlizer: ║                                                                     ║
run pipeline stage %s
intervitens-mini-magnum-5180-v7-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nemo-community-2c-v1
Waiting for inference service chaiml-nemo-community-2c-v1 to be ready
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nothingiisreal-mn-12b-st-5165-v4-mkmlizer
Waiting for job on nothingiisreal-mn-12b-st-5165-v4-mkmlizer to finish
marinaraspaghetti-nemom-1739-v10-mkmlizer: Downloaded to shared memory in 32.397s
marinaraspaghetti-nemom-1739-v10-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmplhmc8e7b, device:0
marinaraspaghetti-nemom-1739-v10-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║     _____            __           __                                ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║       /___/                                                         ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║                                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Version: 0.11.12                                                   ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  https://mk1.ai                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║                                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  The license key for the current software has been verified as      ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  belonging to:                                                      ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║                                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Chai Research Corp.                                                ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║                                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
intervitens-mini-magnum-5180-v7-mkmlizer: Downloaded to shared memory in 29.056s
intervitens-mini-magnum-5180-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpf3ygg_pc, device:0
intervitens-mini-magnum-5180-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
marinaraspaghetti-nemom-1739-v10-mkmlizer: quantized model in 37.902s
marinaraspaghetti-nemom-1739-v10-mkmlizer: Processed model MarinaraSpaghetti/NemoMix-Unleashed-12B in 70.299s
marinaraspaghetti-nemom-1739-v10-mkmlizer: creating bucket guanaco-mkml-models
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: Downloaded to shared memory in 30.190s
marinaraspaghetti-nemom-1739-v10-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmph5jq9gtw, device:0
marinaraspaghetti-nemom-1739-v10-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/config.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/special_tokens_map.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/tokenizer_config.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/tokenizer.json
intervitens-mini-magnum-5180-v7-mkmlizer: quantized model in 37.115s
intervitens-mini-magnum-5180-v7-mkmlizer: Processed model intervitens/mini-magnum-12b-v1.1 in 66.171s
intervitens-mini-magnum-5180-v7-mkmlizer: creating bucket guanaco-mkml-models
intervitens-mini-magnum-5180-v7-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
intervitens-mini-magnum-5180-v7-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/config.json
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/special_tokens_map.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/flywheel_model.0.safetensors
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/tokenizer_config.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:17,  3.04s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:53,  1.22it/s]
Loading 0:   3%|▎         | 11/363 [00:06<02:08,  2.74it/s]
Loading 0:   4%|▍         | 15/363 [00:06<01:22,  4.23it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:43,  7.88it/s]
Loading 0:   7%|▋         | 27/363 [00:06<00:31, 10.76it/s]
Loading 0:   9%|▊         | 31/363 [00:06<00:24, 13.41it/s]
Loading 0:  10%|▉         | 35/363 [00:06<00:20, 15.93it/s]
Loading 0:  11%|█         | 40/363 [00:07<00:20, 15.75it/s]
Loading 0:  12%|█▏        | 43/363 [00:07<00:18, 16.85it/s]
Loading 0:  13%|█▎        | 49/363 [00:07<00:13, 22.83it/s]
Loading 0:  15%|█▍        | 53/363 [00:07<00:12, 24.59it/s]
Loading 0:  16%|█▌        | 58/363 [00:07<00:10, 28.12it/s]
Loading 0:  17%|█▋        | 62/363 [00:07<00:10, 28.98it/s]
Loading 0:  18%|█▊        | 67/363 [00:08<00:09, 31.95it/s]
Loading 0:  20%|█▉        | 71/363 [00:08<00:09, 31.71it/s]
Loading 0:  21%|██        | 76/363 [00:08<00:08, 35.53it/s]
Loading 0:  22%|██▏       | 80/363 [00:08<00:08, 34.31it/s]
Loading 0:  23%|██▎       | 85/363 [00:08<00:07, 37.22it/s]
Loading 0:  25%|██▍       | 89/363 [00:08<00:07, 35.68it/s]
Loading 0:  26%|██▌       | 94/363 [00:08<00:07, 38.23it/s]
Loading 0:  27%|██▋       | 98/363 [00:08<00:07, 36.16it/s]
Loading 0:  28%|██▊       | 103/363 [00:09<00:06, 38.12it/s]
Loading 0:  29%|██▉       | 107/363 [00:09<00:07, 35.86it/s]
Loading 0:  31%|███       | 111/363 [00:09<00:06, 36.29it/s]
Loading 0:  32%|███▏      | 115/363 [00:09<00:07, 34.91it/s]
Loading 0:  33%|███▎      | 121/363 [00:09<00:08, 28.59it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:07, 30.00it/s]
Loading 0:  36%|███▌      | 131/363 [00:09<00:06, 35.34it/s]
Loading 0:  38%|███▊      | 137/363 [00:10<00:06, 36.26it/s]
Loading 0:  39%|███▉      | 141/363 [00:10<00:06, 33.11it/s]
Loading 0:  41%|████      | 148/363 [00:10<00:05, 40.82it/s]
Loading 0:  42%|████▏     | 153/363 [00:10<00:05, 41.74it/s]
Loading 0:  44%|████▎     | 158/363 [00:10<00:04, 41.45it/s]
Loading 0:  45%|████▍     | 163/363 [00:10<00:04, 41.16it/s]
Loading 0:  46%|████▋     | 168/363 [00:10<00:05, 32.84it/s]
Loading 0:  48%|████▊     | 175/363 [00:11<00:04, 38.92it/s]
Loading 0:  50%|████▉     | 180/363 [00:11<00:04, 38.94it/s]
Loading 0:  51%|█████     | 185/363 [00:11<00:04, 39.35it/s]
Loading 0:  52%|█████▏    | 190/363 [00:11<00:04, 41.22it/s]
Loading 0:  54%|█████▎    | 195/363 [00:11<00:04, 34.18it/s]
Loading 0:  55%|█████▌    | 200/363 [00:11<00:04, 36.78it/s]
Loading 0:  56%|█████▌    | 204/363 [00:12<00:06, 24.36it/s]
Loading 0:  58%|█████▊    | 211/363 [00:12<00:04, 31.28it/s]
Loading 0:  59%|█████▉    | 215/363 [00:12<00:04, 31.07it/s]
Loading 0:  61%|██████    | 220/363 [00:12<00:04, 33.52it/s]
Loading 0:  62%|██████▏   | 224/363 [00:12<00:04, 33.05it/s]
Loading 0:  63%|██████▎   | 229/363 [00:12<00:03, 35.98it/s]
Loading 0:  64%|██████▍   | 233/363 [00:12<00:03, 34.38it/s]
Loading 0:  66%|██████▌   | 238/363 [00:12<00:03, 35.43it/s]
Loading 0:  67%|██████▋   | 242/363 [00:13<00:03, 33.95it/s]
Loading 0:  68%|██████▊   | 247/363 [00:13<00:03, 35.78it/s]
Loading 0:  69%|██████▉   | 251/363 [00:13<00:03, 34.17it/s]
Loading 0:  71%|███████   | 256/363 [00:13<00:02, 36.39it/s]
Loading 0:  72%|███████▏  | 260/363 [00:13<00:02, 34.91it/s]
Loading 0:  73%|███████▎  | 265/363 [00:13<00:02, 37.63it/s]
Loading 0:  74%|███████▍  | 269/363 [00:13<00:02, 34.99it/s]
Loading 0:  75%|███████▌  | 273/363 [00:13<00:02, 36.20it/s]
Loading 0:  76%|███████▋  | 277/363 [00:14<00:02, 32.22it/s]
Loading 0:  78%|███████▊  | 283/363 [00:14<00:03, 26.29it/s]
Loading 0:  79%|███████▉  | 286/363 [00:14<00:02, 26.59it/s]
Loading 0:  80%|████████  | 292/363 [00:14<00:02, 33.66it/s]
Loading 0:  82%|████████▏ | 296/363 [00:14<00:01, 35.05it/s]
Loading 0:  83%|████████▎ | 302/363 [00:14<00:01, 39.54it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:01, 40.17it/s]
Loading 0:  86%|████████▌ | 313/363 [00:15<00:01, 40.63it/s]
Loading 0:  88%|████████▊ | 320/363 [00:15<00:00, 45.78it/s]
Loading 0:  90%|████████▉ | 326/363 [00:15<00:00, 43.85it/s]
Loading 0:  91%|█████████ | 331/363 [00:15<00:00, 42.15it/s]
Loading 0:  93%|█████████▎| 338/363 [00:15<00:00, 46.66it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:00, 44.58it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 42.22it/s]
Loading 0:  98%|█████████▊| 355/363 [00:15<00:00, 44.63it/s]
Loading 0:  99%|█████████▉| 360/363 [00:16<00:00, 40.42it/s]
                                                            
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/tokenizer.json
Job marinaraspaghetti-nemom-1739-v10-mkmlizer completed after 107.81s with status: succeeded
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/flywheel_model.0.safetensors
Stopping job with name marinaraspaghetti-nemom-1739-v10-mkmlizer
intervitens-mini-magnum-5180-v7-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 32.47it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 52.04it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 43.38it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 41.37it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 46.41it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:07, 45.49it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:07, 45.43it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:07, 43.16it/s]
Loading 0:  14%|█▍        | 51/363 [00:01<00:07, 41.57it/s]
Loading 0:  16%|█▌        | 58/363 [00:01<00:06, 48.44it/s]
Loading 0:  18%|█▊        | 64/363 [00:01<00:09, 30.07it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 38.12it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:07, 37.99it/s]
Loading 0:  23%|██▎       | 83/363 [00:02<00:07, 38.60it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:06, 41.19it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 41.65it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:06, 42.48it/s]
Loading 0:  29%|██▊       | 104/363 [00:02<00:05, 44.17it/s]
Loading 0:  30%|███       | 109/363 [00:02<00:05, 45.29it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:06, 38.56it/s]
Loading 0:  33%|███▎      | 119/363 [00:02<00:06, 38.05it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:05, 41.27it/s]
Loading 0:  36%|███▌      | 130/363 [00:03<00:05, 39.93it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 41.12it/s]
Loading 0:  39%|███▉      | 141/363 [00:03<00:05, 40.48it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:07, 30.34it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:07, 30.30it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 36.31it/s]
Loading 0:  44%|████▍     | 161/363 [00:04<00:05, 38.23it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:05, 39.14it/s]
Loading 0:  47%|████▋     | 171/363 [00:04<00:04, 41.18it/s]
Loading 0:  48%|████▊     | 176/363 [00:04<00:05, 35.26it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 42.35it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:04, 42.21it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:04, 42.28it/s]
Loading 0:  55%|█████▍    | 198/363 [00:04<00:03, 43.22it/s]
Loading 0:  56%|█████▌    | 203/363 [00:05<00:04, 35.84it/s]
Loading 0:  58%|█████▊    | 210/363 [00:05<00:03, 42.33it/s]
Loading 0:  59%|█████▉    | 215/363 [00:05<00:03, 41.57it/s]
Loading 0:  61%|██████    | 220/363 [00:05<00:03, 43.33it/s]
Loading 0:  62%|██████▏   | 225/363 [00:05<00:05, 27.48it/s]
Loading 0:  63%|██████▎   | 230/363 [00:05<00:04, 29.91it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 37.19it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 38.20it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:03, 38.26it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 40.77it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:02, 35.84it/s]
Loading 0:  73%|███████▎  | 265/363 [00:06<00:02, 44.02it/s]
Loading 0:  75%|███████▍  | 271/363 [00:06<00:02, 42.09it/s]
Loading 0:  76%|███████▌  | 276/363 [00:06<00:02, 42.00it/s]
Loading 0:  78%|███████▊  | 283/363 [00:07<00:01, 46.95it/s]
Loading 0:  79%|███████▉  | 288/363 [00:07<00:01, 47.17it/s]
Loading 0:  81%|████████  | 293/363 [00:07<00:01, 38.27it/s]
Loading 0:  82%|████████▏ | 299/363 [00:07<00:01, 42.56it/s]
Loading 0:  84%|████████▎ | 304/363 [00:14<00:23,  2.51it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:17,  3.22it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:12,  4.11it/s]
Loading 0:  88%|████████▊ | 319/363 [00:14<00:06,  6.46it/s]
Loading 0:  89%|████████▉ | 323/363 [00:14<00:04,  8.05it/s]
Loading 0:  91%|█████████ | 329/363 [00:15<00:03, 11.29it/s]
Loading 0:  92%|█████████▏| 334/363 [00:15<00:01, 14.51it/s]
Loading 0:  93%|█████████▎| 339/363 [00:15<00:01, 16.59it/s]
Loading 0:  95%|█████████▌| 346/363 [00:15<00:00, 22.56it/s]
Loading 0:  97%|█████████▋| 351/363 [00:15<00:00, 25.35it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 28.26it/s]
Loading 0:  99%|█████████▉| 361/363 [00:15<00:00, 31.50it/s]
                                                            
Pipeline stage MKMLizer completed in 108.84s
Job intervitens-mini-magnum-5180-v7-mkmlizer completed after 88.88s with status: succeeded
run pipeline stage %s
Stopping job with name intervitens-mini-magnum-5180-v7-mkmlizer
Running pipeline stage MKMLTemplater
Pipeline stage MKMLizer completed in 90.17s
run pipeline stage %s
Pipeline stage MKMLTemplater completed in 0.38s
Running pipeline stage MKMLTemplater
run pipeline stage %s
Running pipeline stage MKMLDeployer
Pipeline stage MKMLTemplater completed in 0.34s
Creating inference service marinaraspaghetti-nemom-1739-v10
run pipeline stage %s
Running pipeline stage MKMLDeployer
Waiting for inference service marinaraspaghetti-nemom-1739-v10 to be ready
Creating inference service intervitens-mini-magnum-5180-v7
Waiting for inference service intervitens-mini-magnum-5180-v7 to be ready
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: quantized model in 36.133s
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: Processed model nothingiisreal/MN-12B-Starcannon-v2 in 66.323s
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: creating bucket guanaco-mkml-models
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/special_tokens_map.json
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/config.json
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/tokenizer_config.json
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/merges.txt s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/merges.txt
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/vocab.json s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/vocab.json
Inference service chaiml-nemo-community-2a-v1 ready after 140.35990500450134s
Pipeline stage MKMLDeployer completed in 141.12s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0557141304016113s
Received healthy response to inference request in 1.712167501449585s
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/flywheel_model.0.safetensors
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:06,  3.01s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:49,  1.23it/s]
Loading 0:   4%|▎         | 13/363 [00:06<01:43,  3.38it/s]
Loading 0:   5%|▍         | 17/363 [00:06<01:09,  4.95it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:45,  7.55it/s]
Loading 0:   7%|▋         | 27/363 [00:06<00:31, 10.55it/s]
Loading 0:   9%|▉         | 32/363 [00:06<00:23, 14.27it/s]
Loading 0:  10%|█         | 37/363 [00:06<00:17, 18.41it/s]
Loading 0:  12%|█▏        | 42/363 [00:07<00:19, 16.58it/s]
Loading 0:  13%|█▎        | 49/363 [00:07<00:13, 23.27it/s]
Loading 0:  15%|█▍        | 54/363 [00:07<00:11, 26.88it/s]
Loading 0:  16%|█▋        | 59/363 [00:07<00:10, 30.12it/s]
Loading 0:  18%|█▊        | 65/363 [00:07<00:09, 32.67it/s]
Loading 0:  19%|█▉        | 70/363 [00:07<00:08, 34.19it/s]
Loading 0:  21%|██        | 76/363 [00:07<00:07, 38.84it/s]
Loading 0:  22%|██▏       | 81/363 [00:08<00:07, 39.81it/s]
Loading 0:  24%|██▎       | 86/363 [00:08<00:06, 41.73it/s]
Loading 0:  25%|██▌       | 91/363 [00:08<00:06, 41.54it/s]
Loading 0:  26%|██▋       | 96/363 [00:08<00:07, 34.55it/s]
Loading 0:  28%|██▊       | 103/363 [00:08<00:06, 42.11it/s]
Loading 0:  30%|██▉       | 108/363 [00:08<00:06, 42.03it/s]
Loading 0:  31%|███       | 113/363 [00:08<00:05, 43.32it/s]
Loading 0:  33%|███▎      | 118/363 [00:08<00:05, 44.73it/s]
Loading 0:  34%|███▍      | 123/363 [00:09<00:08, 27.45it/s]
Loading 0:  36%|███▌      | 130/363 [00:09<00:06, 33.94it/s]
Loading 0:  37%|███▋      | 135/363 [00:09<00:06, 34.82it/s]
Loading 0:  39%|███▊      | 140/363 [00:09<00:05, 37.42it/s]
Loading 0:  40%|████      | 146/363 [00:09<00:05, 38.09it/s]
Loading 0:  42%|████▏     | 151/363 [00:09<00:05, 39.01it/s]
Loading 0:  44%|████▎     | 158/363 [00:10<00:04, 44.84it/s]
Loading 0:  45%|████▍     | 163/363 [00:10<00:04, 45.48it/s]
Loading 0:  46%|████▋     | 168/363 [00:10<00:04, 39.15it/s]
Loading 0:  48%|████▊     | 176/363 [00:10<00:03, 47.20it/s]
Loading 0:  50%|█████     | 182/363 [00:10<00:04, 43.73it/s]
Loading 0:  52%|█████▏    | 187/363 [00:10<00:04, 41.80it/s]
Loading 0:  53%|█████▎    | 193/363 [00:10<00:03, 45.75it/s]
Loading 0:  55%|█████▍    | 198/363 [00:10<00:03, 44.75it/s]
Loading 0:  56%|█████▌    | 203/363 [00:11<00:04, 33.13it/s]
Loading 0:  57%|█████▋    | 207/363 [00:11<00:04, 33.85it/s]
Loading 0:  58%|█████▊    | 212/363 [00:11<00:04, 36.01it/s]
Loading 0:  60%|█████▉    | 216/363 [00:11<00:04, 36.71it/s]
Loading 0:  61%|██████    | 221/363 [00:11<00:03, 38.78it/s]
Loading 0:  63%|██████▎   | 227/363 [00:11<00:03, 39.72it/s]
Loading 0:  64%|██████▍   | 232/363 [00:11<00:03, 39.83it/s]
Loading 0:  66%|██████▌   | 238/363 [00:12<00:02, 42.44it/s]
Loading 0:  67%|██████▋   | 243/363 [00:12<00:02, 42.36it/s]
Loading 0:  68%|██████▊   | 248/363 [00:12<00:02, 43.33it/s]
Loading 0:  70%|██████▉   | 254/363 [00:12<00:02, 42.16it/s]
Loading 0:  71%|███████▏  | 259/363 [00:12<00:02, 38.92it/s]
Loading 0:  73%|███████▎  | 265/363 [00:12<00:02, 41.92it/s]
Loading 0:  74%|███████▍  | 270/363 [00:12<00:02, 40.50it/s]
Loading 0:  76%|███████▌  | 275/363 [00:12<00:02, 38.98it/s]
Loading 0:  77%|███████▋  | 279/363 [00:13<00:02, 38.16it/s]
Loading 0:  78%|███████▊  | 283/363 [00:13<00:02, 27.93it/s]
Loading 0:  79%|███████▉  | 287/363 [00:13<00:02, 28.87it/s]
Loading 0:  80%|████████  | 292/363 [00:13<00:02, 32.47it/s]
Loading 0:  82%|████████▏ | 296/363 [00:13<00:01, 33.96it/s]
Loading 0:  83%|████████▎ | 302/363 [00:13<00:01, 39.26it/s]
Loading 0:  85%|████████▍ | 307/363 [00:13<00:01, 41.26it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:01, 33.97it/s]
Loading 0:  88%|████████▊ | 319/363 [00:14<00:01, 40.79it/s]
Loading 0:  89%|████████▉ | 324/363 [00:14<00:00, 40.82it/s]
Loading 0:  91%|█████████ | 329/363 [00:14<00:00, 41.28it/s]
Loading 0:  92%|█████████▏| 334/363 [00:14<00:00, 42.10it/s]
Loading 0:  93%|█████████▎| 339/363 [00:14<00:00, 35.45it/s]
Loading 0:  95%|█████████▌| 346/363 [00:14<00:00, 41.37it/s]
Loading 0:  97%|█████████▋| 351/363 [00:14<00:00, 41.61it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 42.68it/s]
Loading 0:  99%|█████████▉| 361/363 [00:15<00:00, 43.74it/s]
                                                            
Job nothingiisreal-mn-12b-st-5165-v4-mkmlizer completed after 89.19s with status: succeeded
Stopping job with name nothingiisreal-mn-12b-st-5165-v4-mkmlizer
Pipeline stage MKMLizer completed in 90.44s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.32s
Received healthy response to inference request in 2.1781928539276123s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service nothingiisreal-mn-12b-st-5165-v4
Waiting for inference service nothingiisreal-mn-12b-st-5165-v4 to be ready
Received healthy response to inference request in 1.531585693359375s
Received healthy response to inference request in 1.497974157333374s
5 requests
0 failed requests
5th percentile: 1.5046964645385743
10th percentile: 1.5114187717437744
20th percentile: 1.5248633861541747
30th percentile: 1.567702054977417
40th percentile: 1.639934778213501
50th percentile: 1.712167501449585
60th percentile: 1.8495861530303954
70th percentile: 1.9870048046112059
80th percentile: 2.0802098751068114
90th percentile: 2.129201364517212
95th percentile: 2.153697109222412
99th percentile: 2.1732937049865724
mean time: 1.7951268672943115
Pipeline stage StressChecker completed in 14.82s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Inference service chaiml-nemo-community-2b-v1 ready after 130.36827421188354s
Pipeline stage MKMLDeployer completed in 131.27s
run pipeline stage %s
Running pipeline stage StressChecker
Pipeline stage TriggerMKMLProfilingPipeline completed in 4.67s
Shutdown handler de-registered
chaiml-nemo-community-2a_v1 status is now deployed due to DeploymentManager action
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Inference service chaiml-nemo-community-2c-v1 ready after 140.34534311294556s
Pipeline stage MKMLDeployer completed in 141.25s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.597785234451294s
Received healthy response to inference request in 1.703284740447998s
Received healthy response to inference request in 1.5114796161651611s
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.6783995628356934s
Received healthy response to inference request in 1.6709704399108887s
5 requests
0 failed requests
5th percentile: 1.5433777809143066
10th percentile: 1.5752759456634522
20th percentile: 1.6390722751617433
30th percentile: 1.6724562644958496
40th percentile: 1.6754279136657715
50th percentile: 1.6783995628356934
60th percentile: 1.6883536338806153
70th percentile: 1.698307704925537
80th percentile: 1.8821848392486573
90th percentile: 2.239985036849976
95th percentile: 2.4188851356506347
99th percentile: 2.5620052146911623
mean time: 1.832383918762207
Pipeline stage StressChecker completed in 14.36s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.88s
Shutdown handler de-registered
chaiml-nemo-community-2c_v1 status is now deployed due to DeploymentManager action
chaiml-nemo-community-2c_v1 status is now inactive due to auto deactivation removed underperforming models
Tearing down inference service chaiml-lexical-nemov8-1k1e5-v11
Deleting key arushimgupta-final-check-3178-v2/special_tokens_map.json from bucket guanaco-mkml-models
Checking if service chaiml-llama-8b-big-retu-8570-v2 is running
Cleaning model data from model cache
Deleting key arushimgupta-lora-save-1-v1/config.json from bucket guanaco-mkml-models
Running pipeline stage MKMLDeleter
Deleting key arushimgupta-final-check-3580-v1/special_tokens_map.json from bucket guanaco-mkml-models
Running pipeline stage MKMLModelDeleter
run pipeline stage %s
Running pipeline stage MKMLModelDeleter
Shutdown handler de-registered
Shutdown handler not registered because Python interpreter is not running in the main thread
Deleting key arushimgupta-final-check-3580-v3/special_tokens_map.json from bucket guanaco-mkml-models
Service chaiml-lexical-nemov8-1k1e5-v11 has been torndown
Service chaiml-lexical-nemov8-1k1e5-v11 has been torndown
Deleting key arushimgupta-final-check-3178-v2/tokenizer.json from bucket guanaco-mkml-models
Deleting key arushimgupta-lora-save-2-v1/config.json from bucket guanaco-mkml-models
Deleting key arushimgupta-lora-save-1-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key arushimgupta-final-check-3580-v1/tokenizer.json from bucket guanaco-mkml-models
Checking if service chaiml-nemo-chai-4bio-me-9462-v2 is running
Tearing down inference service chaiml-llama-8b-big-retu-8570-v2
Cleaning model data from S3
Running pipeline stage MKMLDeleter
Cleaning model data from S3
run pipeline %s
arushimgupta-final-check_2833_v1 status is now torndown due to DeploymentManager action
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of chaiml-nemo-community-5_v1
Pipeline stage MKMLDeleter completed in 17.03s
Deleting key arushimgupta-final-check-3178-v2/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key arushimgupta-lora-save-2-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key arushimgupta-final-check-3580-v1/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key arushimgupta-lora-save-1-v1/special_tokens_map.json from bucket guanaco-mkml-models
Service chaiml-llama-8b-big-retu-8570-v2 has been torndown
Cleaning model data from model cache
Tearing down inference service chaiml-nemo-chai-4bio-me-9462-v2
Cleaning model data from model cache
Checking if service chaiml-nemo-comm-2abio-m-6915-v1 is running
run pipeline stage %s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Deleting key arushimgupta-final-check-3580-v3/tokenizer_config.json from bucket guanaco-mkml-models
admin requested tearing down of chaiml-nemo-lyra-rica-2b_8403_v1
run pipeline stage %s
Pipeline stage MKMLModelDeleter completed in 29.48s
Pipeline stage MKMLModelDeleter completed in 29.85s
Deleting key arushimgupta-lora-save-1-v1/tokenizer.json from bucket guanaco-mkml-models
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0d6bc5dd-2d6a-43b0-80d1-ed98d993cfc9, 37710ecc-f4bb-46c1-93d6-401568f32ee9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:15:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bedec6b2-10be-456b-b899-43da25d85e6e, 9cad9f56-8ad4-4d5f-9b53-fdb236f60483\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'31f6e8d0-1308-497c-b776-5eed2092fee9, 4dfb3e12-34eb-4650-bb81-7574a3b5b917\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:36 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b626f518-1640-40a7-a977-db69d4019f9e, dbd4ea88-baf8-4905-bae2-9adccfabafc2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:38 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fddf772a-f541-47a5-905a-532fad3117a5, 4e8b0759-cd45-4949-8c05-4673139a3399\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:35:38 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5c8b0f93-790a-405a-9814-8860b41382fe, dfb1a2d9-c3e6-4949-841d-f0dbfdb6285d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:40:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ed417b59-cb26-4a43-9c75-1bf0cf44cdcb, 3ac917a1-deab-4952-9df4-21ef7e8a1c41\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:45:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7da9f5a2-471f-42ad-b935-aac771abe801, c1688a1c-e4f5-48d9-8c9e-9e4a798e04aa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:50:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'309b251f-6578-45c4-b637-497237cdd8b7, 0f22f32b-1b19-437f-a331-5e78b0d80b1b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:55:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1e0ee1c0-7e4d-44ba-9429-66493c241fb9, 8bd8f703-4ce9-4b52-b9e9-1be36ccbeab3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:00:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8ea16c14-55c0-45bc-b86f-fb8c10fa3962, d2df502c-9a26-440b-9682-843055227205\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:05:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3a681c2d-5fe4-4200-b70e-609b266bbdc7, f0957715-2c96-457e-94de-e8ca29c997d9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:10:42 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4e88999d-ac0d-4009-94ee-17600498cc8a, c98abb3c-0ebe-4645-a4cd-23c7bff7de1f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:15:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'90d8d268-1239-48ec-b32e-527fc1f1b0d5, 905fc83a-ba4d-44c9-9343-1ec61ed02907\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:20:44 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'28f9e851-6c8d-4e65-a478-e8a3385be83c, 457dcb57-b8bc-4b7d-afe7-0ecd5bc4e279\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:25:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3f2b8b57-31cb-476b-a53c-fc39895e8c0e, 9a771d70-6016-4d5e-9de2-0843bfdf2162\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:30:46 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'07fbdec7-80bc-4a67-a408-97202ae27035, d1933ffe-7715-4dc6-9f41-a2ae367083ff\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:35:44 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cf56f824-8825-48bb-8155-b878b1cebb84, d85d8df9-b196-4ad9-bd7f-64f710e4ae17\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:40:48 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4a66a130-a32b-44fa-97d5-5ba9eb48dbdc, 3389da86-5d28-4cce-8220-8a380dd5536b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:45:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c690ef72-7c29-4318-a0ba-2795e57013ff, 5181c928-891a-49da-9b62-94f0bd0b6f71\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:50:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'34866aac-5fd2-446b-b284-543299324428, a05a3442-fd7c-471f-829a-0692970162b2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:55:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9d92e491-1f84-4d5f-8054-0fed4aa9c598, ba6c0332-f7c4-4cbf-8075-e838b96ee171\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:03 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5c4ee18e-9e90-476f-98e6-2dfd4e433d7e, df609660-a53e-450b-93e1-2ff52bc74370\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:05:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8a7ff050-7289-467b-a56e-ab635d9849fc, 23ab2780-3806-4dcb-a938-0711fb169466\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:10:53 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'54366c16-e8ff-4a06-98a8-4ba67f540a39, c4022c29-4da3-4af5-a455-470318ef0a6e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:15:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c8c4c408-9191-4e1a-ba05-6d9e082e2c2d, 6cc6efbf-5f53-41df-a67c-2523e580cb77\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:20:53 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-community-2c_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-community-2c-v1 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5dfc4537-c166-4e90-b604-b67fce2cda00, 456cc075-afa1-4f8e-ae16-bb20e62a6de9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:25:52 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
chaiml-nemo-community-2c_v1 status is now torndown due to DeploymentManager action