nothingiisreal-mn-12b-st_5165

developer_uid: azuruce
submission_id: nothingiisreal-mn-12b-st_5165_v4
model_name: starcannon
model_group: nothingiisreal/MN-12B-St
status: torndown
timestamp: 2024-09-30T22:43:49+00:00
num_battles: 3861
num_wins: 2006
celo_rating: 1268.62
family_friendly_score: 0.5601420328450954
family_friendly_standard_error: 0.008082857231957716
submission_type: basic
model_repo: nothingiisreal/MN-12B-Starcannon-v2
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: starcannon
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: nothingiisreal/MN-12B-Starcannon-v2
model_size: 13B
ranking_group: single
us_pacific_date: 2024-09-30
win_ratio: 0.5195545195545196
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>', '<|end_of_text|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nothingiisreal-mn-12b-st-5165-v4-mkmlizer
Waiting for job on nothingiisreal-mn-12b-st-5165-v4-mkmlizer to finish
marinaraspaghetti-nemom-1739-v10-mkmlizer: Downloaded to shared memory in 32.397s
marinaraspaghetti-nemom-1739-v10-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmplhmc8e7b, device:0
marinaraspaghetti-nemom-1739-v10-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║     _____            __           __                                ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║       /___/                                                         ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║                                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Version: 0.11.12                                                   ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  https://mk1.ai                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║                                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  The license key for the current software has been verified as      ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  belonging to:                                                      ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║                                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Chai Research Corp.                                                ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ║                                                                     ║
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
intervitens-mini-magnum-5180-v7-mkmlizer: Downloaded to shared memory in 29.056s
intervitens-mini-magnum-5180-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpf3ygg_pc, device:0
intervitens-mini-magnum-5180-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
marinaraspaghetti-nemom-1739-v10-mkmlizer: quantized model in 37.902s
marinaraspaghetti-nemom-1739-v10-mkmlizer: Processed model MarinaraSpaghetti/NemoMix-Unleashed-12B in 70.299s
marinaraspaghetti-nemom-1739-v10-mkmlizer: creating bucket guanaco-mkml-models
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: Downloaded to shared memory in 30.190s
marinaraspaghetti-nemom-1739-v10-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmph5jq9gtw, device:0
marinaraspaghetti-nemom-1739-v10-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/config.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/special_tokens_map.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/tokenizer_config.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/tokenizer.json
intervitens-mini-magnum-5180-v7-mkmlizer: quantized model in 37.115s
intervitens-mini-magnum-5180-v7-mkmlizer: Processed model intervitens/mini-magnum-12b-v1.1 in 66.171s
intervitens-mini-magnum-5180-v7-mkmlizer: creating bucket guanaco-mkml-models
intervitens-mini-magnum-5180-v7-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
intervitens-mini-magnum-5180-v7-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/config.json
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/special_tokens_map.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/marinaraspaghetti-nemom-1739-v10/flywheel_model.0.safetensors
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/tokenizer_config.json
marinaraspaghetti-nemom-1739-v10-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:17,  3.04s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:53,  1.22it/s]
Loading 0:   3%|▎         | 11/363 [00:06<02:08,  2.74it/s]
Loading 0:   4%|▍         | 15/363 [00:06<01:22,  4.23it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:43,  7.88it/s]
Loading 0:   7%|▋         | 27/363 [00:06<00:31, 10.76it/s]
Loading 0:   9%|▊         | 31/363 [00:06<00:24, 13.41it/s]
Loading 0:  10%|▉         | 35/363 [00:06<00:20, 15.93it/s]
Loading 0:  11%|█         | 40/363 [00:07<00:20, 15.75it/s]
Loading 0:  12%|█▏        | 43/363 [00:07<00:18, 16.85it/s]
Loading 0:  13%|█▎        | 49/363 [00:07<00:13, 22.83it/s]
Loading 0:  15%|█▍        | 53/363 [00:07<00:12, 24.59it/s]
Loading 0:  16%|█▌        | 58/363 [00:07<00:10, 28.12it/s]
Loading 0:  17%|█▋        | 62/363 [00:07<00:10, 28.98it/s]
Loading 0:  18%|█▊        | 67/363 [00:08<00:09, 31.95it/s]
Loading 0:  20%|█▉        | 71/363 [00:08<00:09, 31.71it/s]
Loading 0:  21%|██        | 76/363 [00:08<00:08, 35.53it/s]
Loading 0:  22%|██▏       | 80/363 [00:08<00:08, 34.31it/s]
Loading 0:  23%|██▎       | 85/363 [00:08<00:07, 37.22it/s]
Loading 0:  25%|██▍       | 89/363 [00:08<00:07, 35.68it/s]
Loading 0:  26%|██▌       | 94/363 [00:08<00:07, 38.23it/s]
Loading 0:  27%|██▋       | 98/363 [00:08<00:07, 36.16it/s]
Loading 0:  28%|██▊       | 103/363 [00:09<00:06, 38.12it/s]
Loading 0:  29%|██▉       | 107/363 [00:09<00:07, 35.86it/s]
Loading 0:  31%|███       | 111/363 [00:09<00:06, 36.29it/s]
Loading 0:  32%|███▏      | 115/363 [00:09<00:07, 34.91it/s]
Loading 0:  33%|███▎      | 121/363 [00:09<00:08, 28.59it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:07, 30.00it/s]
Loading 0:  36%|███▌      | 131/363 [00:09<00:06, 35.34it/s]
Loading 0:  38%|███▊      | 137/363 [00:10<00:06, 36.26it/s]
Loading 0:  39%|███▉      | 141/363 [00:10<00:06, 33.11it/s]
Loading 0:  41%|████      | 148/363 [00:10<00:05, 40.82it/s]
Loading 0:  42%|████▏     | 153/363 [00:10<00:05, 41.74it/s]
Loading 0:  44%|████▎     | 158/363 [00:10<00:04, 41.45it/s]
Loading 0:  45%|████▍     | 163/363 [00:10<00:04, 41.16it/s]
Loading 0:  46%|████▋     | 168/363 [00:10<00:05, 32.84it/s]
Loading 0:  48%|████▊     | 175/363 [00:11<00:04, 38.92it/s]
Loading 0:  50%|████▉     | 180/363 [00:11<00:04, 38.94it/s]
Loading 0:  51%|█████     | 185/363 [00:11<00:04, 39.35it/s]
Loading 0:  52%|█████▏    | 190/363 [00:11<00:04, 41.22it/s]
Loading 0:  54%|█████▎    | 195/363 [00:11<00:04, 34.18it/s]
Loading 0:  55%|█████▌    | 200/363 [00:11<00:04, 36.78it/s]
Loading 0:  56%|█████▌    | 204/363 [00:12<00:06, 24.36it/s]
Loading 0:  58%|█████▊    | 211/363 [00:12<00:04, 31.28it/s]
Loading 0:  59%|█████▉    | 215/363 [00:12<00:04, 31.07it/s]
Loading 0:  61%|██████    | 220/363 [00:12<00:04, 33.52it/s]
Loading 0:  62%|██████▏   | 224/363 [00:12<00:04, 33.05it/s]
Loading 0:  63%|██████▎   | 229/363 [00:12<00:03, 35.98it/s]
Loading 0:  64%|██████▍   | 233/363 [00:12<00:03, 34.38it/s]
Loading 0:  66%|██████▌   | 238/363 [00:12<00:03, 35.43it/s]
Loading 0:  67%|██████▋   | 242/363 [00:13<00:03, 33.95it/s]
Loading 0:  68%|██████▊   | 247/363 [00:13<00:03, 35.78it/s]
Loading 0:  69%|██████▉   | 251/363 [00:13<00:03, 34.17it/s]
Loading 0:  71%|███████   | 256/363 [00:13<00:02, 36.39it/s]
Loading 0:  72%|███████▏  | 260/363 [00:13<00:02, 34.91it/s]
Loading 0:  73%|███████▎  | 265/363 [00:13<00:02, 37.63it/s]
Loading 0:  74%|███████▍  | 269/363 [00:13<00:02, 34.99it/s]
Loading 0:  75%|███████▌  | 273/363 [00:13<00:02, 36.20it/s]
Loading 0:  76%|███████▋  | 277/363 [00:14<00:02, 32.22it/s]
Loading 0:  78%|███████▊  | 283/363 [00:14<00:03, 26.29it/s]
Loading 0:  79%|███████▉  | 286/363 [00:14<00:02, 26.59it/s]
Loading 0:  80%|████████  | 292/363 [00:14<00:02, 33.66it/s]
Loading 0:  82%|████████▏ | 296/363 [00:14<00:01, 35.05it/s]
Loading 0:  83%|████████▎ | 302/363 [00:14<00:01, 39.54it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:01, 40.17it/s]
Loading 0:  86%|████████▌ | 313/363 [00:15<00:01, 40.63it/s]
Loading 0:  88%|████████▊ | 320/363 [00:15<00:00, 45.78it/s]
Loading 0:  90%|████████▉ | 326/363 [00:15<00:00, 43.85it/s]
Loading 0:  91%|█████████ | 331/363 [00:15<00:00, 42.15it/s]
Loading 0:  93%|█████████▎| 338/363 [00:15<00:00, 46.66it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:00, 44.58it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 42.22it/s]
Loading 0:  98%|█████████▊| 355/363 [00:15<00:00, 44.63it/s]
Loading 0:  99%|█████████▉| 360/363 [00:16<00:00, 40.42it/s]
                                                            
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/tokenizer.json
Job marinaraspaghetti-nemom-1739-v10-mkmlizer completed after 107.81s with status: succeeded
intervitens-mini-magnum-5180-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/intervitens-mini-magnum-5180-v7/flywheel_model.0.safetensors
Stopping job with name marinaraspaghetti-nemom-1739-v10-mkmlizer
intervitens-mini-magnum-5180-v7-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 32.47it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 52.04it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 43.38it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 41.37it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 46.41it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:07, 45.49it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:07, 45.43it/s]
Loading 0:  13%|█▎        | 46/363 [00:01<00:07, 43.16it/s]
Loading 0:  14%|█▍        | 51/363 [00:01<00:07, 41.57it/s]
Loading 0:  16%|█▌        | 58/363 [00:01<00:06, 48.44it/s]
Loading 0:  18%|█▊        | 64/363 [00:01<00:09, 30.07it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 38.12it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:07, 37.99it/s]
Loading 0:  23%|██▎       | 83/363 [00:02<00:07, 38.60it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:06, 41.19it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 41.65it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:06, 42.48it/s]
Loading 0:  29%|██▊       | 104/363 [00:02<00:05, 44.17it/s]
Loading 0:  30%|███       | 109/363 [00:02<00:05, 45.29it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:06, 38.56it/s]
Loading 0:  33%|███▎      | 119/363 [00:02<00:06, 38.05it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:05, 41.27it/s]
Loading 0:  36%|███▌      | 130/363 [00:03<00:05, 39.93it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 41.12it/s]
Loading 0:  39%|███▉      | 141/363 [00:03<00:05, 40.48it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:07, 30.34it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:07, 30.30it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 36.31it/s]
Loading 0:  44%|████▍     | 161/363 [00:04<00:05, 38.23it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:05, 39.14it/s]
Loading 0:  47%|████▋     | 171/363 [00:04<00:04, 41.18it/s]
Loading 0:  48%|████▊     | 176/363 [00:04<00:05, 35.26it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 42.35it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:04, 42.21it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:04, 42.28it/s]
Loading 0:  55%|█████▍    | 198/363 [00:04<00:03, 43.22it/s]
Loading 0:  56%|█████▌    | 203/363 [00:05<00:04, 35.84it/s]
Loading 0:  58%|█████▊    | 210/363 [00:05<00:03, 42.33it/s]
Loading 0:  59%|█████▉    | 215/363 [00:05<00:03, 41.57it/s]
Loading 0:  61%|██████    | 220/363 [00:05<00:03, 43.33it/s]
Loading 0:  62%|██████▏   | 225/363 [00:05<00:05, 27.48it/s]
Loading 0:  63%|██████▎   | 230/363 [00:05<00:04, 29.91it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 37.19it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 38.20it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:03, 38.26it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 40.77it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:02, 35.84it/s]
Loading 0:  73%|███████▎  | 265/363 [00:06<00:02, 44.02it/s]
Loading 0:  75%|███████▍  | 271/363 [00:06<00:02, 42.09it/s]
Loading 0:  76%|███████▌  | 276/363 [00:06<00:02, 42.00it/s]
Loading 0:  78%|███████▊  | 283/363 [00:07<00:01, 46.95it/s]
Loading 0:  79%|███████▉  | 288/363 [00:07<00:01, 47.17it/s]
Loading 0:  81%|████████  | 293/363 [00:07<00:01, 38.27it/s]
Loading 0:  82%|████████▏ | 299/363 [00:07<00:01, 42.56it/s]
Loading 0:  84%|████████▎ | 304/363 [00:14<00:23,  2.51it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:17,  3.22it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:12,  4.11it/s]
Loading 0:  88%|████████▊ | 319/363 [00:14<00:06,  6.46it/s]
Loading 0:  89%|████████▉ | 323/363 [00:14<00:04,  8.05it/s]
Loading 0:  91%|█████████ | 329/363 [00:15<00:03, 11.29it/s]
Loading 0:  92%|█████████▏| 334/363 [00:15<00:01, 14.51it/s]
Loading 0:  93%|█████████▎| 339/363 [00:15<00:01, 16.59it/s]
Loading 0:  95%|█████████▌| 346/363 [00:15<00:00, 22.56it/s]
Loading 0:  97%|█████████▋| 351/363 [00:15<00:00, 25.35it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 28.26it/s]
Loading 0:  99%|█████████▉| 361/363 [00:15<00:00, 31.50it/s]
                                                            
Pipeline stage MKMLizer completed in 108.84s
Job intervitens-mini-magnum-5180-v7-mkmlizer completed after 88.88s with status: succeeded
run pipeline stage %s
Stopping job with name intervitens-mini-magnum-5180-v7-mkmlizer
Running pipeline stage MKMLTemplater
Pipeline stage MKMLizer completed in 90.17s
run pipeline stage %s
Pipeline stage MKMLTemplater completed in 0.38s
Running pipeline stage MKMLTemplater
run pipeline stage %s
Running pipeline stage MKMLDeployer
Pipeline stage MKMLTemplater completed in 0.34s
Creating inference service marinaraspaghetti-nemom-1739-v10
run pipeline stage %s
Running pipeline stage MKMLDeployer
Waiting for inference service marinaraspaghetti-nemom-1739-v10 to be ready
Creating inference service intervitens-mini-magnum-5180-v7
Waiting for inference service intervitens-mini-magnum-5180-v7 to be ready
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: quantized model in 36.133s
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: Processed model nothingiisreal/MN-12B-Starcannon-v2 in 66.323s
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: creating bucket guanaco-mkml-models
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/special_tokens_map.json
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/config.json
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/tokenizer_config.json
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/merges.txt s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/merges.txt
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/vocab.json s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/vocab.json
Inference service chaiml-nemo-community-2a-v1 ready after 140.35990500450134s
Pipeline stage MKMLDeployer completed in 141.12s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0557141304016113s
Received healthy response to inference request in 1.712167501449585s
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nothingiisreal-mn-12b-st-5165-v4/flywheel_model.0.safetensors
nothingiisreal-mn-12b-st-5165-v4-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:06,  3.01s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:49,  1.23it/s]
Loading 0:   4%|▎         | 13/363 [00:06<01:43,  3.38it/s]
Loading 0:   5%|▍         | 17/363 [00:06<01:09,  4.95it/s]
Loading 0:   6%|▌         | 22/363 [00:06<00:45,  7.55it/s]
Loading 0:   7%|▋         | 27/363 [00:06<00:31, 10.55it/s]
Loading 0:   9%|▉         | 32/363 [00:06<00:23, 14.27it/s]
Loading 0:  10%|█         | 37/363 [00:06<00:17, 18.41it/s]
Loading 0:  12%|█▏        | 42/363 [00:07<00:19, 16.58it/s]
Loading 0:  13%|█▎        | 49/363 [00:07<00:13, 23.27it/s]
Loading 0:  15%|█▍        | 54/363 [00:07<00:11, 26.88it/s]
Loading 0:  16%|█▋        | 59/363 [00:07<00:10, 30.12it/s]
Loading 0:  18%|█▊        | 65/363 [00:07<00:09, 32.67it/s]
Loading 0:  19%|█▉        | 70/363 [00:07<00:08, 34.19it/s]
Loading 0:  21%|██        | 76/363 [00:07<00:07, 38.84it/s]
Loading 0:  22%|██▏       | 81/363 [00:08<00:07, 39.81it/s]
Loading 0:  24%|██▎       | 86/363 [00:08<00:06, 41.73it/s]
Loading 0:  25%|██▌       | 91/363 [00:08<00:06, 41.54it/s]
Loading 0:  26%|██▋       | 96/363 [00:08<00:07, 34.55it/s]
Loading 0:  28%|██▊       | 103/363 [00:08<00:06, 42.11it/s]
Loading 0:  30%|██▉       | 108/363 [00:08<00:06, 42.03it/s]
Loading 0:  31%|███       | 113/363 [00:08<00:05, 43.32it/s]
Loading 0:  33%|███▎      | 118/363 [00:08<00:05, 44.73it/s]
Loading 0:  34%|███▍      | 123/363 [00:09<00:08, 27.45it/s]
Loading 0:  36%|███▌      | 130/363 [00:09<00:06, 33.94it/s]
Loading 0:  37%|███▋      | 135/363 [00:09<00:06, 34.82it/s]
Loading 0:  39%|███▊      | 140/363 [00:09<00:05, 37.42it/s]
Loading 0:  40%|████      | 146/363 [00:09<00:05, 38.09it/s]
Loading 0:  42%|████▏     | 151/363 [00:09<00:05, 39.01it/s]
Loading 0:  44%|████▎     | 158/363 [00:10<00:04, 44.84it/s]
Loading 0:  45%|████▍     | 163/363 [00:10<00:04, 45.48it/s]
Loading 0:  46%|████▋     | 168/363 [00:10<00:04, 39.15it/s]
Loading 0:  48%|████▊     | 176/363 [00:10<00:03, 47.20it/s]
Loading 0:  50%|█████     | 182/363 [00:10<00:04, 43.73it/s]
Loading 0:  52%|█████▏    | 187/363 [00:10<00:04, 41.80it/s]
Loading 0:  53%|█████▎    | 193/363 [00:10<00:03, 45.75it/s]
Loading 0:  55%|█████▍    | 198/363 [00:10<00:03, 44.75it/s]
Loading 0:  56%|█████▌    | 203/363 [00:11<00:04, 33.13it/s]
Loading 0:  57%|█████▋    | 207/363 [00:11<00:04, 33.85it/s]
Loading 0:  58%|█████▊    | 212/363 [00:11<00:04, 36.01it/s]
Loading 0:  60%|█████▉    | 216/363 [00:11<00:04, 36.71it/s]
Loading 0:  61%|██████    | 221/363 [00:11<00:03, 38.78it/s]
Loading 0:  63%|██████▎   | 227/363 [00:11<00:03, 39.72it/s]
Loading 0:  64%|██████▍   | 232/363 [00:11<00:03, 39.83it/s]
Loading 0:  66%|██████▌   | 238/363 [00:12<00:02, 42.44it/s]
Loading 0:  67%|██████▋   | 243/363 [00:12<00:02, 42.36it/s]
Loading 0:  68%|██████▊   | 248/363 [00:12<00:02, 43.33it/s]
Loading 0:  70%|██████▉   | 254/363 [00:12<00:02, 42.16it/s]
Loading 0:  71%|███████▏  | 259/363 [00:12<00:02, 38.92it/s]
Loading 0:  73%|███████▎  | 265/363 [00:12<00:02, 41.92it/s]
Loading 0:  74%|███████▍  | 270/363 [00:12<00:02, 40.50it/s]
Loading 0:  76%|███████▌  | 275/363 [00:12<00:02, 38.98it/s]
Loading 0:  77%|███████▋  | 279/363 [00:13<00:02, 38.16it/s]
Loading 0:  78%|███████▊  | 283/363 [00:13<00:02, 27.93it/s]
Loading 0:  79%|███████▉  | 287/363 [00:13<00:02, 28.87it/s]
Loading 0:  80%|████████  | 292/363 [00:13<00:02, 32.47it/s]
Loading 0:  82%|████████▏ | 296/363 [00:13<00:01, 33.96it/s]
Loading 0:  83%|████████▎ | 302/363 [00:13<00:01, 39.26it/s]
Loading 0:  85%|████████▍ | 307/363 [00:13<00:01, 41.26it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:01, 33.97it/s]
Loading 0:  88%|████████▊ | 319/363 [00:14<00:01, 40.79it/s]
Loading 0:  89%|████████▉ | 324/363 [00:14<00:00, 40.82it/s]
Loading 0:  91%|█████████ | 329/363 [00:14<00:00, 41.28it/s]
Loading 0:  92%|█████████▏| 334/363 [00:14<00:00, 42.10it/s]
Loading 0:  93%|█████████▎| 339/363 [00:14<00:00, 35.45it/s]
Loading 0:  95%|█████████▌| 346/363 [00:14<00:00, 41.37it/s]
Loading 0:  97%|█████████▋| 351/363 [00:14<00:00, 41.61it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 42.68it/s]
Loading 0:  99%|█████████▉| 361/363 [00:15<00:00, 43.74it/s]
                                                            
Job nothingiisreal-mn-12b-st-5165-v4-mkmlizer completed after 89.19s with status: succeeded
Stopping job with name nothingiisreal-mn-12b-st-5165-v4-mkmlizer
Pipeline stage MKMLizer completed in 90.44s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.32s
Received healthy response to inference request in 2.1781928539276123s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service nothingiisreal-mn-12b-st-5165-v4
Waiting for inference service nothingiisreal-mn-12b-st-5165-v4 to be ready
Received healthy response to inference request in 1.531585693359375s
Received healthy response to inference request in 1.497974157333374s
5 requests
0 failed requests
5th percentile: 1.5046964645385743
10th percentile: 1.5114187717437744
20th percentile: 1.5248633861541747
30th percentile: 1.567702054977417
40th percentile: 1.639934778213501
50th percentile: 1.712167501449585
60th percentile: 1.8495861530303954
70th percentile: 1.9870048046112059
80th percentile: 2.0802098751068114
90th percentile: 2.129201364517212
95th percentile: 2.153697109222412
99th percentile: 2.1732937049865724
mean time: 1.7951268672943115
Pipeline stage StressChecker completed in 14.82s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Inference service chaiml-nemo-community-2b-v1 ready after 130.36827421188354s
Pipeline stage MKMLDeployer completed in 131.27s
run pipeline stage %s
Running pipeline stage StressChecker
Pipeline stage TriggerMKMLProfilingPipeline completed in 4.67s
Shutdown handler de-registered
chaiml-nemo-community-2a_v1 status is now deployed due to DeploymentManager action
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Inference service chaiml-nemo-community-2c-v1 ready after 140.34534311294556s
Pipeline stage MKMLDeployer completed in 141.25s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.597785234451294s
Received healthy response to inference request in 1.703284740447998s
Received healthy response to inference request in 1.5114796161651611s
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.6783995628356934s
Received healthy response to inference request in 1.6709704399108887s
5 requests
0 failed requests
5th percentile: 1.5433777809143066
10th percentile: 1.5752759456634522
20th percentile: 1.6390722751617433
30th percentile: 1.6724562644958496
40th percentile: 1.6754279136657715
50th percentile: 1.6783995628356934
60th percentile: 1.6883536338806153
70th percentile: 1.698307704925537
80th percentile: 1.8821848392486573
90th percentile: 2.239985036849976
95th percentile: 2.4188851356506347
99th percentile: 2.5620052146911623
mean time: 1.832383918762207
Pipeline stage StressChecker completed in 14.36s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.88s
Shutdown handler de-registered
chaiml-nemo-community-2c_v1 status is now deployed due to DeploymentManager action
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 20.405511093139648
10th percentile: 20.407138729095458
20th percentile: 20.41039400100708
30th percentile: 20.424617290496826
40th percentile: 20.449808597564697
50th percentile: 20.47499990463257
60th percentile: 20.47588481903076
70th percentile: 20.476769733428956
80th percentile: 20.56205334663391
90th percentile: 20.73173565864563
95th percentile: 20.81657681465149
99th percentile: 20.884449739456176
mean time: 20.53390703201294
%s, retrying in %s seconds...
Inference service marinaraspaghetti-nemom-1739-v10 ready after 140.35246992111206s
Pipeline stage MKMLDeployer completed in 141.60s
run pipeline stage %s
Inference service intervitens-mini-magnum-5180-v7 ready after 140.33790850639343s
Running pipeline stage StressChecker
Pipeline stage MKMLDeployer completed in 141.59s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2772250175476074s
Received healthy response to inference request in 1.8627393245697021s
Received healthy response to inference request in 1.613600730895996s
Received healthy response to inference request in 1.7613575458526611s
Received healthy response to inference request in 1.539470911026001s
Received healthy response to inference request in 1.840815544128418s
Received healthy response to inference request in 1.8105297088623047s
Received healthy response to inference request in 1.8727962970733643s
Received healthy response to inference request in 1.5205559730529785s
5 requests
0 failed requests
5th percentile: 1.524338960647583
Received healthy response to inference request in 1.5500566959381104s
10th percentile: 1.5281219482421875
5 requests
20th percentile: 1.5356879234313965
0 failed requests
30th percentile: 1.554296875
5th percentile: 1.5923168659210205
40th percentile: 1.583948802947998
10th percentile: 1.6345770359039307
50th percentile: 1.613600730895996
20th percentile: 1.719097375869751
60th percentile: 1.6923723220825195
30th percentile: 1.7772491455078125
70th percentile: 1.771143913269043
40th percentile: 1.8090323448181151
80th percentile: 1.9038687705993653
50th percentile: 1.840815544128418
90th percentile: 2.090546894073486
60th percentile: 1.8495850563049316
95th percentile: 2.183885955810547
70th percentile: 1.8583545684814453
99th percentile: 2.258557205200195
80th percentile: 1.8647507190704347
mean time: 1.7522764682769776
90th percentile: 1.8687735080718995
Pipeline stage StressChecker completed in 13.90s
95th percentile: 1.8707849025726317
run pipeline stage %s
99th percentile: 1.8723940181732177
Running pipeline stage TriggerMKMLProfilingPipeline
mean time: 1.7775530815124512
run_pipeline:run_in_cloud %s
Pipeline stage StressChecker completed in 14.07s
starting trigger_guanaco_pipeline args=%s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.09s
Shutdown handler de-registered
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.30s
marinaraspaghetti-nemom_1739_v10 status is now deployed due to DeploymentManager action
Shutdown handler de-registered
intervitens-mini-magnum-_5180_v7 status is now deployed due to DeploymentManager action
Inference service nothingiisreal-mn-12b-st-5165-v4 ready after 140.3748984336853s
Pipeline stage MKMLDeployer completed in 141.46s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9871528148651123s
Received healthy response to inference request in 1.6784992218017578s
Received healthy response to inference request in 1.7467222213745117s
Received healthy response to inference request in 1.5273916721343994s
Received healthy response to inference request in 1.4911267757415771s
5 requests
0 failed requests
5th percentile: 1.4983797550201416
10th percentile: 1.5056327342987061
20th percentile: 1.520138692855835
30th percentile: 1.5576131820678711
40th percentile: 1.6180562019348144
50th percentile: 1.6784992218017578
60th percentile: 1.7057884216308594
70th percentile: 1.733077621459961
80th percentile: 1.7948083400726318
90th percentile: 1.8909805774688722
95th percentile: 1.9390666961669922
99th percentile: 1.9775355911254884
mean time: 1.6861785411834718
Pipeline stage StressChecker completed in 11.02s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.68s
Shutdown handler de-registered
nothingiisreal-mn-12b-st_5165_v4 status is now deployed due to DeploymentManager action
nothingiisreal-mn-12b-st_5165_v4 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'107913d9-e850-4218-a1eb-14edd5e577fb, 099bd063-273b-43ca-9f83-96a08bd0904a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b160d1f5-a4ce-42ff-82c0-f0f176824e6e, 5671bd2c-993c-440d-962d-36bc13595f93\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'eacd8c9b-827e-4d46-837f-871650511826, 1107f380-9c5f-44ed-9b30-b6825396b73a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4e714107-5ed4-4d54-af76-063f21a5f82b, a8b57b71-4aec-4373-a866-7b4cecb673d4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9f4349a5-e841-4a45-9ea4-bf932bb369d9, 477a9c1d-90df-42fe-a8d2-675de296af0f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:36:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2fa6230a-014a-4c73-a49d-7ea26bebb239, c2fa2859-c99a-4302-80e8-6f0c89719793\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'86e5ff6f-eea4-4a7d-80de-2ba80c79d027, d60b3e6f-2c1f-4dde-b38f-2991ff25ef0f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9ca81f22-8c96-4819-aa55-969df3fd35ec, 89b29332-5ca5-495c-9cac-0788b2ce7acf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:07 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b5381eba-d832-4036-a4f5-e1b3475f339b, 040de2f1-67f1-4ea2-bcd1-d303cf9e214b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'702be216-8e07-4e1e-a948-f6ca1f67b4a8, dc74d4a7-cb07-4514-ada6-86623542974f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:21 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'008e8b90-35c8-4956-84b5-b0a908a9e0cb, 64d3bf63-df62-4603-8b8b-ad79d7e065ea\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:14 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'759331a4-1122-46eb-8741-03bb600767cd, 06d539e6-d76d-4961-bdeb-c7431da872e1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'62a6c4ea-d4d9-43a1-a2c5-3338190f3355, 44595803-40f4-4ed1-aee0-ee6ba6bd8fcc\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'78d1e3e5-bd84-40cf-a3e5-bfffb08c959c, 9d2e3084-f762-4c26-bd81-5aae111694d7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:16 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a41260ef-3815-46ff-a9d1-0369b7b40510, e0d4c9d6-6abe-4734-b2ca-564aacc1c072\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:15 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2d0d0ae-4df3-42ad-9dd2-deb395e79b00, 078acf39-8ec5-40aa-b9ef-c0fbc20081b8\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c35277f0-496d-4315-9abe-a88795345663, 5ea19120-4c3e-4584-853e-c6d7dd5eb99f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:17 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2347e20-6cfc-4b93-bbff-51b60c3f7595, af484465-d42f-473e-bdce-7cc287427ee3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:23 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b67db4f2-c003-453b-b40f-784ef0b6e598, 091f1192-b64d-4625-ae66-974d72ca8e5a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ba217302-e1de-415a-8e2e-dc2289452c17, 3152f108-3006-4628-850b-882f49cf7ee3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b58415d5-46f2-456a-8c4d-95208c247184, fae8764d-d216-4fba-a062-fc4d916900a3\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:25 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'76f20d76-c6a3-47cf-8e8b-ee699b37f907, 5e8b1713-8ea4-4df6-974c-259bb19bc411\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2b9f34b1-3337-4f7d-a8e0-40caca125ddc, 92b70539-bb96-4f27-9b90-015d50434887\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:26 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'984435c0-9d30-4c37-8015-175c215fb2a8, 3717bc3e-2f7e-4f0d-9702-ce33eb29a9fa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'68835a14-d2ce-4039-a4ad-1b32ba4182cc, 1d1f43a6-8301-4837-a301-4d227dc5468f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'77b644d7-3d8c-4602-9b21-e203351e046a, cdc0eb87-fae7-4dff-b3da-791939d3649b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:31 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of nothingiisreal-mn-12b-st_5165_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
%s, retrying in %s seconds...
Checking if service nothingiisreal-mn-12b-st-5165-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0ba146c9-9e88-484c-9a6e-aeee5f6e73b8, 4957ccfb-0fe3-4c27-9dc0-80927e388e26\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:28 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
nothingiisreal-mn-12b-st_5165_v4 status is now torndown due to DeploymentManager action