zonemercy-vingt-deux-v2-1e5

developer_uid: chai_backend_admin
submission_id: zonemercy-vingt-deux-v2-1e5_v29
model_name: temp-6
model_group: zonemercy/Vingt-Deux-v2-
status: torndown
timestamp: 2024-09-27T14:07:39+00:00
num_battles: 3577
num_wins: 1815
celo_rating: 1253.98
family_friendly_score: 0.56552217453505
family_friendly_standard_error: 0.008362899094889827
submission_type: basic
model_repo: zonemercy/Vingt-Deux-v2-1e5
model_architecture: MistralForCausalLM
model_num_parameters: 22247282688.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: temp-6
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: zonemercy/Vingt-Deux-v2-1e5
model_size: 22B
ranking_group: single
us_pacific_date: 2024-09-27
win_ratio: 0.5074084428291865
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': 'Bot: {message}\n', 'user_template': 'User: {message}\n', 'response_template': 'Bot:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-v2-1e5-v29-mkmlizer
Waiting for job on zonemercy-vingt-deux-v2-1e5-v29-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-v6-2e6-v4-mkmlizer
Waiting for job on zonemercy-vingt-deux-v6-2e6-v4-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-gfv5v6-v1-mkmlizer
Shutdown handler not registered because Python interpreter is not running in the main thread
Waiting for job on zonemercy-vingt-deux-gfv5v6-v1-mkmlizer to finish
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer
Waiting for job on zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer to finish
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: Downloaded to shared memory in 104.692s
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp2mugzdaq, device:0
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: Downloaded to shared memory in 92.976s
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpbr_agc1r, device:0
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: Downloaded to shared memory in 106.259s
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpgck1_8i1, device:0
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: Downloaded to shared memory in 96.312s
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp3p41v2ns, device:0
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: quantized model in 45.161s
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: Processed model zonemercy/Vingt-Deux-v2-1e5 in 149.854s
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/config.json
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/tokenizer_config.json
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/special_tokens_map.json
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/tokenizer.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: quantized model in 47.425s
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: Processed model zonemercy/Vingt-Deux-v3-1e5v0 in 140.401s
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: quantized model in 46.853s
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: Processed model zonemercy/Vingt-Deux-v6-2e6 in 153.112s
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/flywheel_model.1.safetensors
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/special_tokens_map.json
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/config.json
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/tokenizer_config.json
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/tokenizer.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/config.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/special_tokens_map.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/tokenizer_config.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/tokenizer.json
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/flywheel_model.0.safetensors
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/507 [00:00<00:12, 39.79it/s]
Loading 0:   2%|▏         | 8/507 [00:00<00:15, 31.62it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:14, 33.48it/s]
Loading 0:   3%|▎         | 16/507 [00:00<00:15, 32.34it/s]
Loading 0:   4%|▍         | 21/507 [00:00<00:13, 35.49it/s]
Loading 0:   5%|▍         | 25/507 [00:00<00:14, 33.57it/s]
Loading 0:   6%|▌         | 29/507 [00:00<00:13, 34.67it/s]
Loading 0:   7%|▋         | 33/507 [00:01<00:15, 31.14it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:14, 33.02it/s]
Loading 0:   8%|▊         | 41/507 [00:01<00:15, 30.32it/s]
Loading 0:   9%|▉         | 47/507 [00:01<00:12, 35.45it/s]
Loading 0:  10%|█         | 51/507 [00:01<00:13, 34.01it/s]
Loading 0:  11%|█         | 55/507 [00:01<00:19, 23.64it/s]
Loading 0:  11%|█▏        | 58/507 [00:01<00:18, 24.67it/s]
Loading 0:  12%|█▏        | 63/507 [00:02<00:15, 28.30it/s]
Loading 0:  13%|█▎        | 67/507 [00:02<00:15, 28.74it/s]
Loading 0:  14%|█▍        | 71/507 [00:02<00:14, 30.92it/s]
Loading 0:  15%|█▍        | 75/507 [00:02<00:13, 32.25it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:13, 32.16it/s]
Loading 0:  17%|█▋        | 85/507 [00:02<00:11, 35.63it/s]
Loading 0:  18%|█▊        | 89/507 [00:02<00:13, 31.63it/s]
Loading 0:  19%|█▊        | 94/507 [00:02<00:11, 35.78it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:12, 31.90it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:11, 35.89it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:12, 31.98it/s]
Loading 0:  22%|██▏       | 112/507 [00:03<00:11, 35.71it/s]
Loading 0:  23%|██▎       | 116/507 [00:03<00:17, 23.00it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:15, 25.64it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:12, 29.72it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:13, 28.11it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:11, 32.42it/s]
Loading 0:  28%|██▊       | 140/507 [00:04<00:12, 30.56it/s]
Loading 0:  29%|██▊       | 145/507 [00:04<00:10, 34.90it/s]
Loading 0:  29%|██▉       | 149/507 [00:04<00:11, 31.92it/s]
Loading 0:  30%|███       | 154/507 [00:04<00:09, 35.97it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:10, 32.35it/s]
Loading 0:  32%|███▏      | 164/507 [00:05<00:09, 36.40it/s]
Loading 0:  33%|███▎      | 169/507 [00:05<00:12, 28.03it/s]
Loading 0:  34%|███▍      | 173/507 [00:05<00:11, 28.71it/s]
Loading 0:  35%|███▍      | 177/507 [00:05<00:11, 27.72it/s]
Loading 0:  36%|███▌      | 181/507 [00:05<00:10, 30.24it/s]
Loading 0:  36%|███▋      | 185/507 [00:05<00:11, 28.69it/s]
Loading 0:  37%|███▋      | 190/507 [00:06<00:09, 33.28it/s]
Loading 0:  38%|███▊      | 194/507 [00:06<00:10, 30.66it/s]
Loading 0:  39%|███▉      | 199/507 [00:06<00:08, 34.95it/s]
Loading 0:  40%|████      | 203/507 [00:06<00:09, 31.86it/s]
Loading 0:  41%|████▏     | 210/507 [00:06<00:07, 38.65it/s]
Loading 0:  42%|████▏     | 215/507 [00:06<00:07, 37.97it/s]
Loading 0:  43%|████▎     | 219/507 [00:06<00:09, 30.46it/s]
Loading 0:  44%|████▍     | 223/507 [00:07<00:08, 32.42it/s]
Loading 0:  45%|████▍     | 227/507 [00:07<00:10, 27.60it/s]
Loading 0:  46%|████▌     | 231/507 [00:07<00:10, 26.96it/s]
Loading 0:  46%|████▋     | 235/507 [00:07<00:09, 29.45it/s]
Loading 0:  47%|████▋     | 239/507 [00:07<00:10, 26.20it/s]
Loading 0:  48%|████▊     | 244/507 [00:07<00:08, 30.75it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:09, 28.39it/s]
Loading 0:  50%|████▉     | 253/507 [00:08<00:07, 32.47it/s]
Loading 0:  51%|█████     | 257/507 [00:08<00:08, 29.96it/s]
Loading 0:  52%|█████▏    | 262/507 [00:08<00:07, 34.50it/s]
Loading 0:  52%|█████▏    | 266/507 [00:08<00:07, 31.21it/s]
Loading 0:  53%|█████▎    | 271/507 [00:08<00:06, 34.97it/s]
Loading 0:  54%|█████▍    | 275/507 [00:08<00:07, 31.79it/s]
Loading 0:  55%|█████▌    | 280/507 [00:08<00:06, 35.88it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:08, 26.52it/s]
Loading 0:  57%|█████▋    | 288/507 [00:09<00:08, 26.15it/s]
Loading 0:  58%|█████▊    | 293/507 [00:09<00:07, 26.93it/s]
Loading 0:  59%|█████▉    | 298/507 [00:09<00:06, 31.43it/s]
Loading 0:  59%|█████▉    | 299/507 [00:20<00:06, 31.43it/s]
Loading 0:  59%|█████▉    | 300/507 [00:24<03:58,  1.15s/it]
Loading 0:  60%|█████▉    | 302/507 [00:24<03:16,  1.05it/s]
Loading 0:  61%|██████    | 307/507 [00:24<01:58,  1.69it/s]
Loading 0:  61%|██████    | 310/507 [00:24<01:29,  2.19it/s]
Loading 0:  62%|██████▏   | 314/507 [00:24<01:01,  3.16it/s]
Loading 0:  63%|██████▎   | 318/507 [00:24<00:42,  4.44it/s]
Loading 0:  64%|██████▎   | 322/507 [00:24<00:30,  6.06it/s]
Loading 0:  64%|██████▍   | 326/507 [00:24<00:22,  8.20it/s]
Loading 0:  65%|██████▌   | 330/507 [00:25<00:17, 10.33it/s]
Loading 0:  66%|██████▌   | 334/507 [00:25<00:12, 13.31it/s]
Loading 0:  67%|██████▋   | 339/507 [00:25<00:09, 17.41it/s]
Loading 0:  68%|██████▊   | 343/507 [00:25<00:10, 15.96it/s]
Loading 0:  68%|██████▊   | 347/507 [00:25<00:09, 17.74it/s]
Loading 0:  69%|██████▉   | 352/507 [00:25<00:06, 22.52it/s]
Loading 0:  70%|███████   | 356/507 [00:26<00:06, 23.45it/s]
Loading 0:  71%|███████   | 361/507 [00:26<00:05, 28.36it/s]
Loading 0:  72%|███████▏  | 365/507 [00:26<00:05, 27.88it/s]
Loading 0:  73%|███████▎  | 370/507 [00:26<00:04, 32.50it/s]
Loading 0:  74%|███████▍  | 374/507 [00:26<00:04, 30.58it/s]
Loading 0:  75%|███████▌  | 381/507 [00:26<00:03, 37.87it/s]
Loading 0:  76%|███████▌  | 386/507 [00:26<00:03, 37.40it/s]
Loading 0:  77%|███████▋  | 391/507 [00:27<00:03, 31.13it/s]
Loading 0:  78%|███████▊  | 395/507 [00:27<00:04, 24.69it/s]
Loading 0:  79%|███████▉  | 401/507 [00:27<00:03, 26.74it/s]
Loading 0:  80%|████████  | 406/507 [00:27<00:03, 30.87it/s]
Loading 0:  81%|████████  | 410/507 [00:27<00:03, 28.76it/s]
Loading 0:  82%|████████▏ | 415/507 [00:27<00:02, 32.74it/s]
Loading 0:  83%|████████▎ | 419/507 [00:28<00:02, 30.41it/s]
Loading 0:  84%|████████▎ | 424/507 [00:28<00:02, 34.64it/s]
Loading 0:  84%|████████▍ | 428/507 [00:28<00:02, 31.46it/s]
Loading 0:  85%|████████▌ | 433/507 [00:28<00:02, 35.08it/s]
Loading 0:  86%|████████▌ | 437/507 [00:28<00:02, 31.79it/s]
Loading 0:  88%|████████▊ | 444/507 [00:28<00:01, 38.57it/s]
Loading 0:  89%|████████▊ | 449/507 [00:28<00:01, 37.82it/s]
Loading 0:  90%|████████▉ | 454/507 [00:29<00:01, 34.88it/s]
Loading 0:  90%|█████████ | 458/507 [00:31<00:08,  6.11it/s]
Loading 0:  91%|█████████ | 461/507 [00:31<00:06,  7.17it/s]
Loading 0:  92%|█████████▏| 465/507 [00:31<00:04,  9.05it/s]
Loading 0:  93%|█████████▎| 470/507 [00:31<00:02, 12.47it/s]
Loading 0:  93%|█████████▎| 474/507 [00:31<00:02, 14.32it/s]
Loading 0:  94%|█████████▍| 479/507 [00:31<00:01, 18.62it/s]
Loading 0:  95%|█████████▌| 483/507 [00:32<00:01, 20.20it/s]
Loading 0:  96%|█████████▋| 488/507 [00:32<00:00, 25.01it/s]
Loading 0:  97%|█████████▋| 492/507 [00:32<00:00, 24.83it/s]
Loading 0:  98%|█████████▊| 497/507 [00:32<00:00, 29.39it/s]
Loading 0:  99%|█████████▉| 501/507 [00:32<00:00, 27.79it/s]
Loading 0: 100%|█████████▉| 506/507 [00:32<00:00, 32.42it/s]
                                                            
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/flywheel_model.1.safetensors
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/flywheel_model.0.safetensors
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/flywheel_model.0.safetensors
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/flywheel_model.1.safetensors
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:22, 21.91it/s]
Loading 0:   2%|▏         | 10/507 [00:00<00:16, 30.48it/s]
Loading 0:   3%|▎         | 14/507 [00:00<00:18, 26.15it/s]
Loading 0:   4%|▎         | 19/507 [00:00<00:15, 30.68it/s]
Loading 0:   5%|▍         | 23/507 [00:00<00:17, 27.34it/s]
Loading 0:   6%|▌         | 28/507 [00:00<00:15, 31.19it/s]
Loading 0:   6%|▋         | 32/507 [00:01<00:17, 27.48it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:15, 30.81it/s]
Loading 0:   8%|▊         | 41/507 [00:01<00:16, 27.45it/s]
Loading 0:   9%|▉         | 46/507 [00:01<00:15, 30.58it/s]
Loading 0:  10%|▉         | 50/507 [00:01<00:16, 27.11it/s]
Loading 0:  10%|█         | 53/507 [00:02<00:24, 18.28it/s]
Loading 0:  11%|█         | 56/507 [00:02<00:25, 17.56it/s]
Loading 0:  12%|█▏        | 61/507 [00:02<00:20, 21.94it/s]
Loading 0:  13%|█▎        | 65/507 [00:02<00:20, 21.55it/s]
Loading 0:  14%|█▍        | 70/507 [00:02<00:17, 25.61it/s]
Loading 0:  14%|█▍        | 73/507 [00:02<00:16, 26.01it/s]
Loading 0:  16%|█▌        | 80/507 [00:03<00:14, 28.90it/s]
Loading 0:  17%|█▋        | 85/507 [00:03<00:13, 32.04it/s]
Loading 0:  18%|█▊        | 89/507 [00:03<00:14, 29.17it/s]
Loading 0:  19%|█▊        | 94/507 [00:03<00:12, 33.06it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:13, 29.77it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:11, 33.72it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:13, 30.55it/s]
Loading 0:  22%|██▏       | 112/507 [00:04<00:11, 34.15it/s]
Loading 0:  23%|██▎       | 116/507 [00:04<00:17, 22.12it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:15, 25.21it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:12, 29.52it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:13, 27.95it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:11, 31.88it/s]
Loading 0:  28%|██▊       | 140/507 [00:05<00:13, 27.93it/s]
Loading 0:  29%|██▊       | 145/507 [00:05<00:11, 31.53it/s]
Loading 0:  29%|██▉       | 149/507 [00:05<00:12, 29.30it/s]
Loading 0:  30%|███       | 154/507 [00:05<00:10, 32.57it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:11, 30.71it/s]
Loading 0:  32%|███▏      | 164/507 [00:05<00:09, 35.11it/s]
Loading 0:  33%|███▎      | 168/507 [00:05<00:09, 35.25it/s]
Loading 0:  34%|███▍      | 172/507 [00:06<00:13, 24.71it/s]
Loading 0:  35%|███▍      | 176/507 [00:06<00:13, 25.26it/s]
Loading 0:  36%|███▌      | 183/507 [00:06<00:09, 32.73it/s]
Loading 0:  37%|███▋      | 187/507 [00:06<00:09, 32.60it/s]
Loading 0:  38%|███▊      | 192/507 [00:06<00:08, 35.25it/s]
Loading 0:  39%|███▊      | 196/507 [00:06<00:09, 33.54it/s]
Loading 0:  40%|███▉      | 201/507 [00:06<00:08, 35.42it/s]
Loading 0:  40%|████      | 205/507 [00:07<00:08, 34.19it/s]
Loading 0:  41%|████▏     | 210/507 [00:07<00:08, 36.63it/s]
Loading 0:  42%|████▏     | 214/507 [00:07<00:08, 33.70it/s]
Loading 0:  43%|████▎     | 218/507 [00:07<00:08, 33.04it/s]
Loading 0:  44%|████▍     | 222/507 [00:07<00:08, 31.95it/s]
Loading 0:  45%|████▍     | 226/507 [00:07<00:12, 22.88it/s]
Loading 0:  45%|████▌     | 230/507 [00:08<00:12, 22.55it/s]
Loading 0:  46%|████▋     | 235/507 [00:08<00:10, 26.58it/s]
Loading 0:  47%|████▋     | 239/507 [00:08<00:10, 25.21it/s]
Loading 0:  48%|████▊     | 244/507 [00:08<00:09, 29.05it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:09, 26.98it/s]
Loading 0:  50%|████▉     | 253/507 [00:08<00:08, 31.07it/s]
Loading 0:  51%|█████     | 257/507 [00:09<00:08, 28.04it/s]
Loading 0:  52%|█████▏    | 262/507 [00:09<00:07, 31.70it/s]
Loading 0:  52%|█████▏    | 266/507 [00:09<00:08, 28.85it/s]
Loading 0:  53%|█████▎    | 271/507 [00:09<00:07, 32.11it/s]
Loading 0:  54%|█████▍    | 275/507 [00:09<00:07, 29.46it/s]
Loading 0:  55%|█████▌    | 280/507 [00:09<00:06, 32.91it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:09, 23.87it/s]
Loading 0:  57%|█████▋    | 287/507 [00:10<00:09, 22.04it/s]
Loading 0:  58%|█████▊    | 293/507 [00:10<00:08, 24.67it/s]
Loading 0:  59%|█████▉    | 298/507 [00:10<00:07, 28.44it/s]
Loading 0:  60%|█████▉    | 302/507 [00:25<03:34,  1.05s/it]
Loading 0:  61%|██████    | 307/507 [00:25<02:24,  1.39it/s]
Loading 0:  61%|██████▏   | 311/507 [00:26<01:46,  1.84it/s]
Loading 0:  63%|██████▎   | 318/507 [00:26<01:02,  3.02it/s]
Loading 0:  64%|██████▎   | 322/507 [00:26<00:47,  3.89it/s]
Loading 0:  64%|██████▍   | 327/507 [00:26<00:33,  5.40it/s]
Loading 0:  65%|██████▌   | 331/507 [00:26<00:25,  6.87it/s]
Loading 0:  66%|██████▌   | 335/507 [00:26<00:19,  8.76it/s]
Loading 0:  67%|██████▋   | 339/507 [00:26<00:15, 11.17it/s]
Loading 0:  68%|██████▊   | 343/507 [00:27<00:14, 11.40it/s]
Loading 0:  68%|██████▊   | 347/507 [00:27<00:11, 13.36it/s]
Loading 0:  69%|██████▉   | 352/507 [00:27<00:08, 17.62it/s]
Loading 0:  70%|███████   | 356/507 [00:27<00:07, 19.22it/s]
Loading 0:  71%|███████   | 361/507 [00:27<00:06, 23.80it/s]
Loading 0:  72%|███████▏  | 365/507 [00:27<00:05, 23.86it/s]
Loading 0:  73%|███████▎  | 370/507 [00:28<00:04, 28.12it/s]
Loading 0:  74%|███████▍  | 374/507 [00:28<00:04, 27.04it/s]
Loading 0:  75%|███████▍  | 379/507 [00:28<00:04, 31.35it/s]
Loading 0:  76%|███████▌  | 383/507 [00:28<00:04, 28.88it/s]
Loading 0:  77%|███████▋  | 388/507 [00:28<00:03, 33.10it/s]
Loading 0:  77%|███████▋  | 392/507 [00:28<00:03, 30.31it/s]
Loading 0:  78%|███████▊  | 396/507 [00:29<00:05, 21.88it/s]
Loading 0:  79%|███████▉  | 401/507 [00:29<00:04, 23.88it/s]
Loading 0:  80%|████████  | 406/507 [00:29<00:03, 26.95it/s]
Loading 0:  81%|████████  | 410/507 [00:29<00:03, 25.58it/s]
Loading 0:  82%|████████▏ | 415/507 [00:29<00:03, 28.81it/s]
Loading 0:  83%|████████▎ | 419/507 [00:29<00:03, 26.72it/s]
Loading 0:  84%|████████▎ | 424/507 [00:30<00:02, 30.29it/s]
Loading 0:  84%|████████▍ | 428/507 [00:30<00:02, 27.95it/s]
Loading 0:  85%|████████▌ | 433/507 [00:30<00:02, 31.48it/s]
Loading 0:  86%|████████▌ | 437/507 [00:30<00:02, 28.72it/s]
Loading 0:  87%|████████▋ | 442/507 [00:30<00:01, 32.62it/s]
Loading 0:  88%|████████▊ | 446/507 [00:30<00:02, 29.59it/s]
Loading 0:  89%|████████▉ | 451/507 [00:30<00:01, 33.52it/s]
Loading 0:  90%|████████▉ | 455/507 [00:33<00:09,  5.57it/s]
Loading 0:  91%|█████████ | 459/507 [00:33<00:06,  7.11it/s]
Loading 0:  92%|█████████▏| 465/507 [00:33<00:04, 10.08it/s]
Loading 0:  93%|█████████▎| 470/507 [00:33<00:02, 13.30it/s]
Loading 0:  93%|█████████▎| 474/507 [00:33<00:02, 14.97it/s]
Loading 0:  94%|█████████▍| 479/507 [00:33<00:01, 18.98it/s]
Loading 0:  95%|█████████▌| 483/507 [00:34<00:01, 20.16it/s]
Loading 0:  96%|█████████▋| 488/507 [00:34<00:00, 24.53it/s]
Loading 0:  97%|█████████▋| 492/507 [00:34<00:00, 24.15it/s]
Loading 0:  98%|█████████▊| 497/507 [00:34<00:00, 28.37it/s]
Loading 0:  99%|█████████▉| 501/507 [00:34<00:00, 26.89it/s]
Loading 0: 100%|█████████▉| 506/507 [00:34<00:00, 31.22it/s]
                                                            
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:20, 24.71it/s]
Loading 0:   2%|▏         | 10/507 [00:00<00:15, 32.85it/s]
Loading 0:   3%|▎         | 14/507 [00:00<00:17, 28.75it/s]
Loading 0:   4%|▎         | 19/507 [00:00<00:14, 34.26it/s]
Loading 0:   5%|▍         | 23/507 [00:00<00:16, 30.12it/s]
Loading 0:   6%|▌         | 28/507 [00:00<00:13, 34.64it/s]
Loading 0:   6%|▋         | 32/507 [00:01<00:15, 30.33it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:13, 33.63it/s]
Loading 0:   8%|▊         | 41/507 [00:01<00:15, 30.07it/s]
Loading 0:   9%|▉         | 46/507 [00:01<00:13, 33.28it/s]
Loading 0:  10%|▉         | 50/507 [00:01<00:14, 30.58it/s]
Loading 0:  11%|█         | 54/507 [00:01<00:21, 21.16it/s]
Loading 0:  11%|█         | 57/507 [00:02<00:21, 21.31it/s]
Loading 0:  12%|█▏        | 61/507 [00:02<00:18, 24.30it/s]
Loading 0:  13%|█▎        | 65/507 [00:02<00:18, 24.08it/s]
Loading 0:  14%|█▍        | 70/507 [00:02<00:15, 28.54it/s]
Loading 0:  15%|█▍        | 75/507 [00:02<00:13, 31.37it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:13, 31.10it/s]
Loading 0:  17%|█▋        | 85/507 [00:02<00:12, 34.40it/s]
Loading 0:  18%|█▊        | 89/507 [00:03<00:13, 30.83it/s]
Loading 0:  19%|█▊        | 94/507 [00:03<00:12, 34.17it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:13, 30.15it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:12, 33.57it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:13, 30.18it/s]
Loading 0:  22%|██▏       | 112/507 [00:03<00:11, 33.20it/s]
Loading 0:  23%|██▎       | 116/507 [00:04<00:18, 21.25it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:15, 24.25it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:13, 28.35it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:14, 26.83it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:12, 30.74it/s]
Loading 0:  28%|██▊       | 140/507 [00:04<00:12, 28.41it/s]
Loading 0:  29%|██▊       | 145/507 [00:04<00:11, 32.03it/s]
Loading 0:  29%|██▉       | 149/507 [00:05<00:12, 29.35it/s]
Loading 0:  30%|███       | 154/507 [00:05<00:10, 32.98it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:11, 29.77it/s]
Loading 0:  32%|███▏      | 163/507 [00:05<00:10, 32.68it/s]
Loading 0:  33%|███▎      | 168/507 [00:05<00:10, 33.48it/s]
Loading 0:  34%|███▍      | 172/507 [00:06<00:14, 22.47it/s]
Loading 0:  35%|███▍      | 176/507 [00:06<00:14, 22.87it/s]
Loading 0:  36%|███▌      | 181/507 [00:06<00:11, 27.25it/s]
Loading 0:  36%|███▋      | 185/507 [00:06<00:12, 26.49it/s]
Loading 0:  37%|███▋      | 190/507 [00:06<00:10, 30.38it/s]
Loading 0:  38%|███▊      | 194/507 [00:06<00:11, 28.12it/s]
Loading 0:  39%|███▉      | 199/507 [00:06<00:09, 32.09it/s]
Loading 0:  40%|████      | 203/507 [00:07<00:10, 29.58it/s]
Loading 0:  41%|████      | 208/507 [00:07<00:08, 33.59it/s]
Loading 0:  42%|████▏     | 212/507 [00:07<00:09, 29.80it/s]
Loading 0:  43%|████▎     | 217/507 [00:07<00:08, 33.74it/s]
Loading 0:  44%|████▎     | 221/507 [00:07<00:09, 30.53it/s]
Loading 0:  44%|████▍     | 225/507 [00:07<00:12, 21.72it/s]
Loading 0:  45%|████▌     | 230/507 [00:08<00:11, 23.94it/s]
Loading 0:  46%|████▋     | 235/507 [00:08<00:09, 28.03it/s]
Loading 0:  47%|████▋     | 239/507 [00:08<00:10, 26.49it/s]
Loading 0:  48%|████▊     | 244/507 [00:08<00:08, 30.31it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:09, 27.61it/s]
Loading 0:  50%|████▉     | 253/507 [00:08<00:08, 31.24it/s]
Loading 0:  51%|█████     | 257/507 [00:08<00:08, 28.30it/s]
Loading 0:  52%|█████▏    | 262/507 [00:09<00:07, 32.27it/s]
Loading 0:  52%|█████▏    | 266/507 [00:09<00:08, 29.78it/s]
Loading 0:  53%|█████▎    | 271/507 [00:09<00:06, 33.74it/s]
Loading 0:  54%|█████▍    | 275/507 [00:09<00:07, 30.45it/s]
Loading 0:  55%|█████▌    | 280/507 [00:09<00:06, 34.60it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:08, 25.29it/s]
Loading 0:  57%|█████▋    | 288/507 [00:10<00:08, 25.28it/s]
Loading 0:  58%|█████▊    | 293/507 [00:10<00:08, 26.29it/s]
Loading 0:  59%|█████▉    | 298/507 [00:10<00:06, 30.54it/s]
Loading 0:  59%|█████▉    | 299/507 [00:25<00:06, 30.54it/s]
Loading 0:  59%|█████▉    | 300/507 [00:25<04:06,  1.19s/it]
Loading 0:  60%|█████▉    | 302/507 [00:25<03:22,  1.01it/s]
Loading 0:  61%|██████    | 307/507 [00:25<02:02,  1.63it/s]
Loading 0:  61%|██████    | 310/507 [00:25<01:32,  2.12it/s]
Loading 0:  62%|██████▏   | 313/507 [00:25<01:08,  2.81it/s]
Loading 0:  62%|██████▏   | 316/507 [00:25<00:51,  3.74it/s]
Loading 0:  63%|██████▎   | 320/507 [00:26<00:35,  5.30it/s]
Loading 0:  64%|██████▍   | 325/507 [00:26<00:22,  8.02it/s]
Loading 0:  65%|██████▍   | 329/507 [00:26<00:17, 10.12it/s]
Loading 0:  66%|██████▌   | 334/507 [00:26<00:12, 13.74it/s]
Loading 0:  67%|██████▋   | 339/507 [00:26<00:09, 17.58it/s]
Loading 0:  68%|██████▊   | 343/507 [00:26<00:10, 15.71it/s]
Loading 0:  68%|██████▊   | 347/507 [00:27<00:09, 17.31it/s]
Loading 0:  69%|██████▉   | 352/507 [00:27<00:07, 21.90it/s]
Loading 0:  70%|███████   | 356/507 [00:27<00:06, 22.74it/s]
Loading 0:  71%|███████   | 361/507 [00:27<00:05, 27.33it/s]
Loading 0:  72%|███████▏  | 365/507 [00:27<00:05, 26.80it/s]
Loading 0:  73%|███████▎  | 370/507 [00:27<00:04, 31.29it/s]
Loading 0:  74%|███████▍  | 374/507 [00:27<00:04, 29.73it/s]
Loading 0:  75%|███████▌  | 381/507 [00:28<00:03, 36.84it/s]
Loading 0:  76%|███████▌  | 386/507 [00:28<00:03, 36.13it/s]
Loading 0:  77%|███████▋  | 390/507 [00:28<00:04, 28.26it/s]
Loading 0:  78%|███████▊  | 394/507 [00:28<00:03, 29.86it/s]
Loading 0:  79%|███████▊  | 398/507 [00:28<00:04, 25.71it/s]
Loading 0:  79%|███████▉  | 401/507 [00:28<00:04, 23.46it/s]
Loading 0:  80%|████████  | 406/507 [00:29<00:03, 28.15it/s]
Loading 0:  81%|████████  | 410/507 [00:29<00:03, 26.57it/s]
Loading 0:  82%|████████▏ | 415/507 [00:29<00:03, 30.49it/s]
Loading 0:  83%|████████▎ | 419/507 [00:29<00:03, 28.08it/s]
Loading 0:  84%|████████▎ | 424/507 [00:29<00:02, 32.06it/s]
Loading 0:  84%|████████▍ | 428/507 [00:29<00:02, 29.51it/s]
Loading 0:  85%|████████▌ | 433/507 [00:29<00:02, 32.91it/s]
Loading 0:  86%|████████▌ | 437/507 [00:30<00:02, 30.15it/s]
Loading 0:  87%|████████▋ | 442/507 [00:30<00:01, 34.38it/s]
Loading 0:  88%|████████▊ | 446/507 [00:30<00:01, 30.95it/s]
Loading 0:  89%|████████▉ | 451/507 [00:30<00:01, 34.98it/s]
Loading 0:  90%|████████▉ | 455/507 [00:32<00:09,  5.66it/s]
Loading 0:  91%|█████████ | 459/507 [00:32<00:06,  7.21it/s]
Loading 0:  92%|█████████▏| 465/507 [00:33<00:04, 10.19it/s]
Loading 0:  93%|█████████▎| 470/507 [00:33<00:02, 13.35it/s]
Loading 0:  93%|█████████▎| 474/507 [00:33<00:02, 14.99it/s]
Loading 0:  94%|█████████▍| 479/507 [00:33<00:01, 18.97it/s]
Loading 0:  95%|█████████▌| 483/507 [00:33<00:01, 19.87it/s]
Loading 0:  96%|█████████▋| 488/507 [00:33<00:00, 24.14it/s]
Loading 0:  97%|█████████▋| 492/507 [00:33<00:00, 23.40it/s]
Loading 0:  98%|█████████▊| 497/507 [00:34<00:00, 25.25it/s]
Loading 0:  99%|█████████▉| 501/507 [00:34<00:00, 24.79it/s]
Loading 0: 100%|█████████▉| 506/507 [00:34<00:00, 29.21it/s]
                                                            
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: quantized model in 47.773s
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: Processed model zonemercy/Vingt-Deux-v1-1e5 in 144.086s
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/config.json
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/special_tokens_map.json
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/tokenizer_config.json
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/tokenizer.json
Job zonemercy-vingt-deux-v2-1e5-v29-mkmlizer completed after 218.2s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v2-1e5-v29-mkmlizer
Pipeline stage MKMLizer completed in 218.88s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.25s
run pipeline stage %s
Job zonemercy-vingt-deux-v6-2e6-v4-mkmlizer completed after 218.38s with status: succeeded
Running pipeline stage MKMLDeployer
Stopping job with name zonemercy-vingt-deux-v6-2e6-v4-mkmlizer
Creating inference service zonemercy-vingt-deux-v2-1e5-v29
Pipeline stage MKMLizer completed in 219.23s
run pipeline stage %s
Waiting for inference service zonemercy-vingt-deux-v2-1e5-v29 to be ready
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.27s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-v6-2e6-v4
Job zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer completed after 218.89s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer
Waiting for inference service zonemercy-vingt-deux-v6-2e6-v4 to be ready
Pipeline stage MKMLizer completed in 219.92s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.28s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-v3-1e5v0-v8
Waiting for inference service zonemercy-vingt-deux-v3-1e5v0-v8 to be ready
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/flywheel_model.1.safetensors
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/flywheel_model.0.safetensors
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:22, 22.72it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:13, 37.78it/s]
Loading 0:   3%|▎         | 17/507 [00:00<00:13, 37.49it/s]
Loading 0:   4%|▍         | 22/507 [00:00<00:12, 38.44it/s]
Loading 0:   5%|▌         | 27/507 [00:00<00:11, 40.42it/s]
Loading 0:   6%|▋         | 32/507 [00:00<00:14, 32.58it/s]
Loading 0:   8%|▊         | 39/507 [00:01<00:11, 39.08it/s]
Loading 0:   9%|▊         | 44/507 [00:01<00:12, 37.88it/s]
Loading 0:  10%|▉         | 49/507 [00:01<00:15, 30.29it/s]
Loading 0:  10%|█         | 53/507 [00:01<00:20, 22.51it/s]
Loading 0:  11%|█         | 56/507 [00:01<00:21, 21.13it/s]
Loading 0:  12%|█▏        | 61/507 [00:02<00:17, 25.71it/s]
Loading 0:  13%|█▎        | 65/507 [00:02<00:18, 24.45it/s]
Loading 0:  14%|█▍        | 70/507 [00:02<00:15, 28.76it/s]
Loading 0:  15%|█▍        | 75/507 [00:02<00:13, 31.09it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:14, 30.07it/s]
Loading 0:  17%|█▋        | 85/507 [00:02<00:12, 33.08it/s]
Loading 0:  18%|█▊        | 89/507 [00:02<00:14, 28.78it/s]
Loading 0:  19%|█▊        | 94/507 [00:03<00:12, 32.04it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:14, 28.52it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:12, 31.76it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:14, 28.54it/s]
Loading 0:  22%|██▏       | 112/507 [00:03<00:12, 31.70it/s]
Loading 0:  23%|██▎       | 116/507 [00:04<00:19, 20.39it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:16, 23.31it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:13, 27.34it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:14, 25.33it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:12, 29.37it/s]
Loading 0:  28%|██▊       | 140/507 [00:04<00:13, 27.36it/s]
Loading 0:  29%|██▊       | 145/507 [00:04<00:11, 31.22it/s]
Loading 0:  29%|██▉       | 149/507 [00:05<00:12, 28.28it/s]
Loading 0:  30%|███       | 154/507 [00:05<00:11, 31.92it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:12, 28.76it/s]
Loading 0:  32%|███▏      | 163/507 [00:05<00:10, 32.85it/s]
Loading 0:  33%|███▎      | 168/507 [00:05<00:10, 33.90it/s]
Loading 0:  34%|███▍      | 172/507 [00:06<00:14, 22.35it/s]
Loading 0:  35%|███▍      | 176/507 [00:06<00:14, 22.50it/s]
Loading 0:  36%|███▌      | 181/507 [00:06<00:12, 26.91it/s]
Loading 0:  36%|███▋      | 185/507 [00:06<00:12, 25.83it/s]
Loading 0:  37%|███▋      | 190/507 [00:06<00:10, 30.07it/s]
Loading 0:  38%|███▊      | 194/507 [00:06<00:11, 27.82it/s]
Loading 0:  39%|███▉      | 199/507 [00:06<00:09, 31.57it/s]
Loading 0:  40%|████      | 203/507 [00:07<00:10, 28.65it/s]
Loading 0:  41%|████      | 208/507 [00:07<00:09, 33.11it/s]
Loading 0:  42%|████▏     | 212/507 [00:07<00:10, 29.43it/s]
Loading 0:  43%|████▎     | 217/507 [00:07<00:08, 33.51it/s]
Loading 0:  44%|████▎     | 221/507 [00:07<00:09, 30.37it/s]
Loading 0:  44%|████▍     | 225/507 [00:07<00:13, 21.68it/s]
Loading 0:  45%|████▌     | 230/507 [00:08<00:11, 23.50it/s]
Loading 0:  46%|████▋     | 235/507 [00:08<00:09, 27.52it/s]
Loading 0:  47%|████▋     | 239/507 [00:08<00:10, 25.62it/s]
Loading 0:  48%|████▊     | 244/507 [00:08<00:08, 29.62it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:09, 27.05it/s]
Loading 0:  50%|████▉     | 253/507 [00:08<00:08, 31.04it/s]
Loading 0:  51%|█████     | 257/507 [00:08<00:08, 28.10it/s]
Loading 0:  52%|█████▏    | 262/507 [00:09<00:07, 31.99it/s]
Loading 0:  52%|█████▏    | 266/507 [00:09<00:08, 28.52it/s]
Loading 0:  53%|█████▎    | 271/507 [00:09<00:07, 31.93it/s]
Loading 0:  54%|█████▍    | 275/507 [00:09<00:08, 28.80it/s]
Loading 0:  55%|█████▌    | 280/507 [00:09<00:07, 32.41it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:09, 24.15it/s]
Loading 0:  57%|█████▋    | 287/507 [00:10<00:10, 21.88it/s]
Loading 0:  58%|█████▊    | 293/507 [00:10<00:08, 24.71it/s]
Loading 0:  59%|█████▉    | 298/507 [00:10<00:07, 28.78it/s]
Loading 0:  60%|█████▉    | 302/507 [00:25<03:34,  1.05s/it]
Loading 0:  61%|██████    | 307/507 [00:25<02:23,  1.39it/s]
Loading 0:  61%|██████▏   | 311/507 [00:26<01:46,  1.84it/s]
Loading 0:  62%|██████▏   | 316/507 [00:26<01:11,  2.68it/s]
Loading 0:  63%|██████▎   | 320/507 [00:26<00:53,  3.53it/s]
Loading 0:  64%|██████▍   | 327/507 [00:26<00:31,  5.67it/s]
Loading 0:  65%|██████▌   | 331/507 [00:26<00:24,  7.06it/s]
Loading 0:  66%|██████▌   | 335/507 [00:26<00:19,  8.86it/s]
Loading 0:  67%|██████▋   | 339/507 [00:26<00:14, 11.21it/s]
Loading 0:  68%|██████▊   | 343/507 [00:27<00:14, 11.47it/s]
Loading 0:  68%|██████▊   | 347/507 [00:27<00:12, 13.30it/s]
Loading 0:  69%|██████▉   | 352/507 [00:27<00:08, 17.46it/s]
Loading 0:  70%|███████   | 356/507 [00:27<00:08, 18.69it/s]
Loading 0:  71%|███████   | 361/507 [00:27<00:06, 23.16it/s]
Loading 0:  72%|███████▏  | 365/507 [00:28<00:06, 23.05it/s]
Loading 0:  73%|███████▎  | 370/507 [00:28<00:05, 27.33it/s]
Loading 0:  74%|███████▍  | 374/507 [00:28<00:05, 26.04it/s]
Loading 0:  75%|███████▍  | 379/507 [00:28<00:04, 30.42it/s]
Loading 0:  76%|███████▌  | 383/507 [00:28<00:04, 27.95it/s]
Loading 0:  77%|███████▋  | 388/507 [00:28<00:03, 32.28it/s]
Loading 0:  77%|███████▋  | 392/507 [00:28<00:03, 29.62it/s]
Loading 0:  78%|███████▊  | 396/507 [00:29<00:05, 21.89it/s]
Loading 0:  79%|███████▉  | 401/507 [00:29<00:04, 23.61it/s]
Loading 0:  80%|████████  | 406/507 [00:29<00:03, 27.65it/s]
Loading 0:  81%|████████  | 410/507 [00:29<00:03, 25.75it/s]
Loading 0:  82%|████████▏ | 415/507 [00:29<00:03, 29.57it/s]
Loading 0:  83%|████████▎ | 419/507 [00:29<00:03, 27.16it/s]
Loading 0:  84%|████████▎ | 424/507 [00:30<00:02, 30.31it/s]
Loading 0:  84%|████████▍ | 428/507 [00:30<00:02, 27.45it/s]
Loading 0:  85%|████████▌ | 433/507 [00:30<00:02, 31.15it/s]
Loading 0:  86%|████████▌ | 437/507 [00:30<00:02, 28.22it/s]
Loading 0:  87%|████████▋ | 442/507 [00:30<00:02, 32.29it/s]
Loading 0:  88%|████████▊ | 446/507 [00:30<00:02, 29.06it/s]
Loading 0:  89%|████████▉ | 451/507 [00:30<00:01, 33.07it/s]
Loading 0:  90%|████████▉ | 455/507 [00:33<00:09,  5.49it/s]
Loading 0:  91%|█████████ | 459/507 [00:33<00:06,  6.97it/s]
Loading 0:  92%|█████████▏| 465/507 [00:33<00:04,  9.85it/s]
Loading 0:  93%|█████████▎| 472/507 [00:33<00:02, 14.33it/s]
Loading 0:  94%|█████████▍| 476/507 [00:33<00:01, 16.33it/s]
Loading 0:  95%|█████████▍| 481/507 [00:34<00:01, 19.90it/s]
Loading 0:  96%|█████████▌| 485/507 [00:34<00:01, 21.68it/s]
Loading 0:  97%|█████████▋| 490/507 [00:34<00:00, 24.89it/s]
Loading 0:  97%|█████████▋| 494/507 [00:34<00:00, 25.06it/s]
Loading 0:  98%|█████████▊| 498/507 [00:34<00:00, 27.72it/s]
Loading 0:  99%|█████████▉| 502/507 [00:34<00:00, 26.15it/s]
Loading 0: 100%|█████████▉| 506/507 [00:34<00:00, 27.60it/s]
                                                            
Job zonemercy-vingt-deux-v1-1e5-v28-mkmlizer completed after 239.14s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v1-1e5-v28-mkmlizer
Pipeline stage MKMLizer completed in 239.79s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.31s
run pipeline stage %s
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: Downloaded to shared memory in 207.378s
Running pipeline stage MKMLDeployer
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp5lk5x204, device:0
Creating inference service zonemercy-vingt-deux-v1-1e5-v28
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Waiting for inference service zonemercy-vingt-deux-v1-1e5-v28 to be ready
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: quantized model in 52.803s
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: Processed model zonemercy/Vingt-Deux-gfv5v6 in 260.181s
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/special_tokens_map.json
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/config.json
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/tokenizer_config.json
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/tokenizer.json
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/flywheel_model.1.safetensors
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/flywheel_model.0.safetensors
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/507 [00:00<00:14, 34.01it/s]
Loading 0:   2%|▏         | 8/507 [00:00<00:17, 27.86it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:18, 27.15it/s]
Loading 0:   3%|▎         | 15/507 [00:00<00:21, 22.96it/s]
Loading 0:   4%|▎         | 19/507 [00:00<00:18, 26.61it/s]
Loading 0:   5%|▍         | 23/507 [00:00<00:16, 29.35it/s]
Loading 0:   5%|▌         | 27/507 [00:01<00:27, 17.76it/s]
Loading 0:   6%|▌         | 31/507 [00:01<00:22, 21.08it/s]
Loading 0:   7%|▋         | 34/507 [00:01<00:21, 21.62it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:20, 23.19it/s]
Loading 0:   8%|▊         | 40/507 [00:01<00:19, 23.61it/s]
Loading 0:   8%|▊         | 43/507 [00:01<00:19, 23.89it/s]
Loading 0:   9%|▉         | 46/507 [00:01<00:18, 25.00it/s]
Loading 0:  10%|▉         | 50/507 [00:02<00:16, 28.01it/s]
Loading 0:  10%|█         | 53/507 [00:02<00:23, 19.65it/s]
Loading 0:  11%|█         | 56/507 [00:02<00:21, 20.99it/s]
Loading 0:  12%|█▏        | 59/507 [00:02<00:25, 17.44it/s]
Loading 0:  13%|█▎        | 64/507 [00:02<00:19, 23.12it/s]
Loading 0:  13%|█▎        | 67/507 [00:02<00:18, 23.43it/s]
Loading 0:  14%|█▍        | 70/507 [00:03<00:18, 23.66it/s]
Loading 0:  14%|█▍        | 73/507 [00:03<00:17, 24.85it/s]
Loading 0:  15%|█▌        | 77/507 [00:03<00:15, 27.44it/s]
Loading 0:  16%|█▌        | 80/507 [00:03<00:21, 19.42it/s]
Loading 0:  16%|█▋        | 83/507 [00:03<00:22, 18.82it/s]
Loading 0:  17%|█▋        | 87/507 [00:03<00:18, 22.53it/s]
Loading 0:  18%|█▊        | 91/507 [00:03<00:16, 25.84it/s]
Loading 0:  19%|█▊        | 94/507 [00:04<00:16, 24.79it/s]
Loading 0:  19%|█▉        | 97/507 [00:04<00:16, 25.16it/s]
Loading 0:  20%|█▉        | 100/507 [00:04<00:20, 19.92it/s]
Loading 0:  21%|██        | 105/507 [00:04<00:15, 25.38it/s]
Loading 0:  21%|██▏       | 108/507 [00:04<00:22, 17.43it/s]
Loading 0:  22%|██▏       | 112/507 [00:04<00:19, 20.74it/s]
Loading 0:  23%|██▎       | 115/507 [00:05<00:18, 21.45it/s]
Loading 0:  23%|██▎       | 118/507 [00:05<00:16, 23.04it/s]
Loading 0:  24%|██▍       | 121/507 [00:05<00:16, 23.50it/s]
Loading 0:  24%|██▍       | 124/507 [00:05<00:16, 22.71it/s]
Loading 0:  25%|██▌       | 127/507 [00:05<00:15, 24.35it/s]
Loading 0:  26%|██▌       | 131/507 [00:05<00:13, 27.12it/s]
Loading 0:  26%|██▋       | 134/507 [00:05<00:19, 18.98it/s]
Loading 0:  27%|██▋       | 137/507 [00:06<00:18, 19.74it/s]
Loading 0:  28%|██▊       | 140/507 [00:06<00:21, 17.04it/s]
Loading 0:  29%|██▊       | 145/507 [00:06<00:16, 22.61it/s]
Loading 0:  29%|██▉       | 148/507 [00:06<00:16, 22.28it/s]
Loading 0:  30%|██▉       | 151/507 [00:06<00:15, 22.79it/s]
Loading 0:  30%|███       | 154/507 [00:06<00:14, 24.17it/s]
Loading 0:  31%|███       | 158/507 [00:06<00:12, 27.09it/s]
Loading 0:  32%|███▏      | 161/507 [00:07<00:17, 19.90it/s]
Loading 0:  32%|███▏      | 164/507 [00:07<00:16, 21.11it/s]
Loading 0:  33%|███▎      | 167/507 [00:07<00:19, 17.76it/s]
Loading 0:  34%|███▍      | 172/507 [00:07<00:14, 23.31it/s]
Loading 0:  35%|███▍      | 175/507 [00:07<00:14, 23.01it/s]
Loading 0:  35%|███▌      | 178/507 [00:07<00:14, 23.21it/s]
Loading 0:  36%|███▌      | 181/507 [00:08<00:13, 24.67it/s]
Loading 0:  36%|███▋      | 185/507 [00:08<00:11, 27.91it/s]
Loading 0:  37%|███▋      | 188/507 [00:08<00:16, 19.28it/s]
Loading 0:  38%|███▊      | 191/507 [00:08<00:15, 20.50it/s]
Loading 0:  38%|███▊      | 194/507 [00:08<00:18, 17.38it/s]
Loading 0:  39%|███▉      | 199/507 [00:08<00:13, 22.86it/s]
Loading 0:  40%|███▉      | 202/507 [00:09<00:12, 23.51it/s]
Loading 0:  40%|████      | 205/507 [00:09<00:12, 23.76it/s]
Loading 0:  41%|████      | 208/507 [00:09<00:12, 24.62it/s]
Loading 0:  42%|████▏     | 212/507 [00:09<00:10, 27.26it/s]
Loading 0:  42%|████▏     | 215/507 [00:09<00:15, 18.97it/s]
Loading 0:  43%|████▎     | 218/507 [00:09<00:14, 20.26it/s]
Loading 0:  44%|████▎     | 221/507 [00:10<00:16, 17.34it/s]
Loading 0:  45%|████▍     | 226/507 [00:10<00:12, 22.85it/s]
Loading 0:  45%|████▌     | 229/507 [00:10<00:11, 23.29it/s]
Loading 0:  46%|████▌     | 232/507 [00:10<00:11, 23.15it/s]
Loading 0:  46%|████▋     | 235/507 [00:10<00:11, 24.42it/s]
Loading 0:  47%|████▋     | 239/507 [00:10<00:09, 27.29it/s]
Loading 0:  48%|████▊     | 242/507 [00:10<00:13, 19.01it/s]
Loading 0:  48%|████▊     | 245/507 [00:11<00:12, 20.39it/s]
Loading 0:  49%|████▉     | 248/507 [00:11<00:14, 17.60it/s]
Loading 0:  50%|████▉     | 253/507 [00:11<00:11, 22.88it/s]
Loading 0:  50%|█████     | 256/507 [00:11<00:10, 23.35it/s]
Loading 0:  51%|█████     | 259/507 [00:11<00:10, 23.63it/s]
Loading 0:  52%|█████▏    | 262/507 [00:11<00:09, 24.93it/s]
Loading 0:  52%|█████▏    | 266/507 [00:11<00:08, 27.94it/s]
Loading 0:  53%|█████▎    | 269/507 [00:12<00:11, 19.87it/s]
Loading 0:  54%|█████▎    | 272/507 [00:12<00:11, 20.97it/s]
Loading 0:  54%|█████▍    | 275/507 [00:12<00:13, 17.69it/s]
Loading 0:  55%|█████▌    | 280/507 [00:12<00:09, 23.32it/s]
Loading 0:  56%|█████▌    | 283/507 [00:12<00:09, 23.95it/s]
Loading 0:  56%|█████▋    | 286/507 [00:12<00:09, 24.15it/s]
Loading 0:  57%|█████▋    | 289/507 [00:12<00:08, 25.50it/s]
Loading 0:  58%|█████▊    | 293/507 [00:13<00:07, 28.23it/s]
Loading 0:  58%|█████▊    | 296/507 [00:13<00:10, 20.29it/s]
Loading 0:  59%|█████▉    | 299/507 [00:13<00:09, 21.40it/s]
Loading 0:  60%|█████▉    | 302/507 [00:27<04:50,  1.42s/it]
Loading 0:  60%|██████    | 305/507 [00:28<03:25,  1.02s/it]
Loading 0:  61%|██████    | 308/507 [00:28<02:26,  1.36it/s]
Loading 0:  61%|██████▏   | 311/507 [00:28<01:46,  1.84it/s]
Loading 0:  62%|██████▏   | 316/507 [00:28<01:02,  3.04it/s]
Loading 0:  63%|██████▎   | 320/507 [00:28<00:43,  4.30it/s]
Loading 0:  64%|██████▎   | 323/507 [00:28<00:36,  5.09it/s]
Loading 0:  64%|██████▍   | 326/507 [00:29<00:28,  6.46it/s]
Loading 0:  65%|██████▍   | 329/507 [00:29<00:23,  7.48it/s]
Loading 0:  66%|██████▌   | 334/507 [00:29<00:15, 11.08it/s]
Loading 0:  66%|██████▋   | 337/507 [00:29<00:13, 12.86it/s]
Loading 0:  67%|██████▋   | 340/507 [00:29<00:11, 14.61it/s]
Loading 0:  68%|██████▊   | 343/507 [00:29<00:09, 16.96it/s]
Loading 0:  68%|██████▊   | 347/507 [00:29<00:07, 20.58it/s]
Loading 0:  69%|██████▉   | 350/507 [00:30<00:09, 16.07it/s]
Loading 0:  70%|██████▉   | 353/507 [00:30<00:08, 17.80it/s]
Loading 0:  70%|███████   | 356/507 [00:30<00:09, 15.82it/s]
Loading 0:  71%|███████   | 361/507 [00:30<00:06, 21.09it/s]
Loading 0:  72%|███████▏  | 364/507 [00:30<00:06, 22.05it/s]
Loading 0:  72%|███████▏  | 367/507 [00:30<00:06, 22.53it/s]
Loading 0:  73%|███████▎  | 370/507 [00:31<00:05, 23.97it/s]
Loading 0:  74%|███████▍  | 374/507 [00:31<00:04, 27.12it/s]
Loading 0:  74%|███████▍  | 377/507 [00:31<00:06, 19.22it/s]
Loading 0:  75%|███████▍  | 380/507 [00:31<00:06, 20.37it/s]
Loading 0:  76%|███████▌  | 383/507 [00:31<00:07, 16.88it/s]
Loading 0:  77%|███████▋  | 388/507 [00:31<00:05, 22.21it/s]
Loading 0:  77%|███████▋  | 391/507 [00:32<00:05, 22.61it/s]
Loading 0:  78%|███████▊  | 394/507 [00:32<00:04, 22.82it/s]
Loading 0:  78%|███████▊  | 397/507 [00:32<00:04, 24.35it/s]
Loading 0:  79%|███████▉  | 401/507 [00:32<00:03, 27.40it/s]
Loading 0:  80%|███████▉  | 404/507 [00:32<00:05, 18.62it/s]
Loading 0:  80%|████████  | 407/507 [00:32<00:05, 19.95it/s]
Loading 0:  81%|████████  | 410/507 [00:33<00:05, 16.69it/s]
Loading 0:  82%|████████▏ | 415/507 [00:33<00:04, 21.94it/s]
Loading 0:  82%|████████▏ | 418/507 [00:33<00:03, 22.41it/s]
Loading 0:  83%|████████▎ | 421/507 [00:33<00:03, 22.70it/s]
Loading 0:  84%|████████▎ | 424/507 [00:33<00:03, 23.45it/s]
Loading 0:  84%|████████▍ | 428/507 [00:33<00:03, 26.32it/s]
Loading 0:  85%|████████▌ | 431/507 [00:34<00:04, 17.88it/s]
Loading 0:  86%|████████▌ | 434/507 [00:34<00:03, 19.48it/s]
Loading 0:  86%|████████▌ | 437/507 [00:34<00:04, 16.83it/s]
Loading 0:  87%|████████▋ | 442/507 [00:34<00:02, 21.93it/s]
Loading 0:  88%|████████▊ | 445/507 [00:34<00:02, 22.44it/s]
Loading 0:  88%|████████▊ | 448/507 [00:34<00:02, 22.81it/s]
Loading 0:  89%|████████▉ | 451/507 [00:34<00:02, 24.42it/s]
Loading 0:  90%|████████▉ | 455/507 [00:34<00:01, 27.36it/s]
Loading 0:  90%|█████████ | 458/507 [00:35<00:02, 18.88it/s]
Loading 0:  91%|█████████ | 461/507 [00:35<00:02, 20.44it/s]
Loading 0:  92%|█████████▏| 464/507 [00:35<00:02, 16.79it/s]
Loading 0:  93%|█████████▎| 469/507 [00:35<00:01, 22.05it/s]
Loading 0:  93%|█████████▎| 472/507 [00:35<00:01, 22.55it/s]
Loading 0:  94%|█████████▎| 475/507 [00:35<00:01, 22.95it/s]
Loading 0:  94%|█████████▍| 478/507 [00:36<00:01, 24.47it/s]
Loading 0:  95%|█████████▌| 482/507 [00:36<00:00, 26.99it/s]
Loading 0:  96%|█████████▌| 485/507 [00:38<00:05,  4.28it/s]
Loading 0:  96%|█████████▋| 488/507 [00:38<00:03,  5.43it/s]
Loading 0:  97%|█████████▋| 491/507 [00:38<00:02,  7.02it/s]
Loading 0:  97%|█████████▋| 494/507 [00:38<00:01,  8.82it/s]
Loading 0:  98%|█████████▊| 497/507 [00:39<00:00, 10.99it/s]
Loading 0:  99%|█████████▊| 500/507 [00:39<00:00, 13.14it/s]
Loading 0:  99%|█████████▉| 503/507 [00:39<00:00, 14.93it/s]
Loading 0: 100%|█████████▉| 506/507 [00:39<00:00, 17.35it/s]
                                                            
Job zonemercy-vingt-deux-gfv5v6-v1-mkmlizer completed after 300.77s with status: succeeded
Stopping job with name zonemercy-vingt-deux-gfv5v6-v1-mkmlizer
Pipeline stage MKMLizer completed in 301.84s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.29s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-gfv5v6-v1
Waiting for inference service zonemercy-vingt-deux-gfv5v6-v1 to be ready
Inference service zonemercy-vingt-deux-v6-2e6-v4 ready after 230.50821113586426s
Pipeline stage MKMLDeployer completed in 231.67s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.1409964561462402s
Received healthy response to inference request in 2.4466605186462402s
Inference service zonemercy-vingt-deux-v2-1e5-v29 ready after 240.5290093421936s
Pipeline stage MKMLDeployer completed in 241.72s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.5495691299438477s
Inference service zonemercy-vingt-deux-v3-1e5v0-v8 ready after 240.5324902534485s
Pipeline stage MKMLDeployer completed in 241.62s
Received healthy response to inference request in 2.3935165405273438s
Received healthy response to inference request in 2.8595969676971436s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.465632200241089s
Received healthy response to inference request in 2.4500367641448975s
5 requests
0 failed requests
5th percentile: 2.404145336151123
Received healthy response to inference request in 2.8449018001556396s
10th percentile: 2.4147741317749025
20th percentile: 2.436031723022461
30th percentile: 2.45045485496521
40th percentile: 2.4580435276031496
50th percentile: 2.465632200241089
60th percentile: 2.4992069721221926
70th percentile: 2.532781744003296
Received healthy response to inference request in 2.3773341178894043s
80th percentile: 2.6678545951843264
90th percentile: 2.904425525665283
95th percentile: 3.0227109909057615
99th percentile: 3.1173393630981443
Received healthy response to inference request in 2.6949636936187744s
mean time: 2.5992749691009522
Pipeline stage StressChecker completed in 18.74s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.3296053409576416s
Received healthy response to inference request in 4.669405221939087s
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.94s
Shutdown handler de-registered
zonemercy-vingt-deux-v6-2e6_v4 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 2.5488362312316895s
Received healthy response to inference request in 2.6253466606140137s
5 requests
Inference service zonemercy-vingt-deux-v1-1e5-v28 ready after 240.64416527748108s
0 failed requests
Pipeline stage MKMLDeployer completed in 241.97s
5th percentile: 2.3391510963439943
run pipeline stage %s
10th percentile: 2.3486968517303466
Running pipeline stage StressChecker
20th percentile: 2.3677883625030516
30th percentile: 2.391874647140503
40th percentile: 2.4209557056427
50th percentile: 2.4500367641448975
60th percentile: 2.4895565509796143
70th percentile: 2.529076337814331
80th percentile: 2.6109883785247803
90th percentile: 2.735292673110962
95th percentile: 2.7974448204040527
99th percentile: 2.8471665382385254
mean time: 2.5130818843841554
Pipeline stage StressChecker completed in 21.64s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
Received healthy response to inference request in 2.9589476585388184s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 4.633564233779907s
5 requests
0 failed requests
5th percentile: 2.639270067214966
10th percentile: 2.653193473815918
20th percentile: 2.6810402870178223
30th percentile: 2.7249513149261473
40th percentile: 2.7849265575408935
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.75s
50th percentile: 2.8449018001556396
Shutdown handler de-registered
Received healthy response to inference request in 2.5984013080596924s
60th percentile: 3.5603667736053466
60th percentile: 3.5603667736053466
zonemercy-vingt-deux-v2-1e5_v29 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.27s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.20s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service zonemercy-vingt-deux-v2-1e5-v29-profiler
Waiting for inference service zonemercy-vingt-deux-v2-1e5-v29-profiler to be ready
Tearing down inference service zonemercy-vingt-deux-v2-1e5-v29-profiler
%s, retrying in %s seconds...
Creating inference service zonemercy-vingt-deux-v2-1e5-v29-profiler
Waiting for inference service zonemercy-vingt-deux-v2-1e5-v29-profiler to be ready
Tearing down inference service zonemercy-vingt-deux-v2-1e5-v29-profiler
%s, retrying in %s seconds...
Creating inference service zonemercy-vingt-deux-v2-1e5-v29-profiler
Ignoring service zonemercy-vingt-deux-v2-1e5-v29-profiler already deployed
Waiting for inference service zonemercy-vingt-deux-v2-1e5-v29-profiler to be ready
Tearing down inference service zonemercy-vingt-deux-v2-1e5-v29-profiler
clean up pipeline due to error=%s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.23s
Shutdown handler de-registered
zonemercy-vingt-deux-v2-1e5_v29 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
admin requested tearing down of zonemercy-vingt-deux-v3-1e5v0_v8
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage MKMLDeleter
run pipeline %s
run pipeline stage %s
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v3-1e5v0-v8 is running
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v3-1e5v0-v8 is running
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v3-1e5v0-v8 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6bbedf65-6a0e-4653-b58e-741510e31336, fca25cbb-a598-4822-875b-15ef24474774\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b3a31c44-909f-4657-b4b2-4f4ca15a6753, 0858ef29-471d-4bf7-97c4-8a8fefecf16c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:35 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0b6f296d-3d82-4ea0-965c-effdb7d762ae, e07eb474-efac-4bb2-8ef2-57f5aabade67\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:18 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9c5c1327-b611-49da-929a-8de35511f501, 4d95ea92-7158-4051-9c1b-1d01c93cc8bf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:22 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b8fa2ea0-ef19-479c-bad1-9f5e1ef2852a, a0e41ff3-2624-4fc5-8913-ee545bcb33d2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dc7687e6-9043-4ee6-a862-eea7416afbae, eecae84e-a613-46f4-91c0-2459335bfe43\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:38:42 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b8a81068-79c4-49d9-bfea-b0973b4c1260, f04292ca-972d-449a-9699-87f49a872a0c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:24 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'19438a25-b21e-4797-ad0d-d700d5e72c16, 804279aa-6dd8-40fe-97fc-7499d93a2020\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'edf4ce19-7ca6-46d6-ae2e-6dc069fd02ea, 5d83b89b-ad70-4f62-a4b0-525a5501530a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:29 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4cf40a17-5296-48a6-a1cc-3d4e4299f6d4, 982e8589-daea-4bee-b327-59a215ea6cb0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:30 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'78a77b53-a236-432d-b937-29e1d8022a09, 2cbac3d2-b2cc-4096-b752-9c10120e65cb\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:46 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'96dfab70-3dc5-4185-bbc9-ca79c1c09147, f2dc6dd0-1f6f-48a6-a20b-1aab62fc5bf9\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:38 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5b543681-be8e-4bc8-a56e-9b8d7db53c37, b2b75e4f-30ae-4067-99be-bbe8b65121a0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:38 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bde1c8bc-123d-42d7-b13b-0357ea091a8e, 037a7c2f-efbf-4714-8105-0689b4757f46\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b71ee44c-2c7f-49be-9336-f25a67f28c25, 5df97c0e-bd64-42e7-a3b2-59f8424af338\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ef355a7a-2ecb-45d7-a94a-cd2e6c8f0ba1, 107bf3c1-f0b1-43a7-be07-743839b25b70\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5703402d-b0fd-4135-965d-79a0f8fe2de3, 1192433c-975a-4a74-8ada-03c1f863cb84\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bc8b8299-76f8-4586-8500-6b3c94dae5dc, 854d6d54-1957-4ef9-8152-b3a7e19eb41a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cdf4660b-055a-43be-b68c-07eb9985b3cb, 231c911c-a517-4b8e-b103-7d4f118a4f1d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:41:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'587a11dc-14c6-4906-8836-f60d6105ce75, 927f55fc-0597-41f7-b19b-2c3aaf7a7b5a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:46:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8f630148-33f3-4682-ab97-09d2851cfe63, 77dd8af6-9908-4cbf-8bd5-e1e45458c6ec\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:51:52 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'017f28c7-17bd-4f95-973f-01bc44c80195, 7dd4ea4b-90bd-41a6-82c9-1e8d33935dba\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:56:54 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a2de6056-e2b3-48d5-8857-e52f30f1294d, d26ed98e-9d74-44bc-8bed-cb182c727682\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:02:08 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a784c8ec-61d5-4b45-9632-8a8d3a07fa1e, 87d3a8d7-a641-44d8-9f71-3f5f1abc8fcc\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:06:54 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0f17fa4c-a590-4bb1-ac8b-8b3b56975d3b, 11d14cf3-9eb6-4f32-b0ea-d0e3d8040de0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:11:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'77a577f5-abe8-4c1c-981b-7252afdf6737, ba652ef4-3058-40f1-a73f-632da096ba19\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:16:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
%s, retrying in %s seconds...
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Checking if service zonemercy-vingt-deux-v1-1e5-v27 is running
Shutdown handler not registered because Python interpreter is not running in the main thread
%s, retrying in %s seconds...
run pipeline %s
Checking if service zonemercy-vingt-deux-v1-1e5-v27 is running
run pipeline stage %s
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c6ac0f73-7e69-4c90-8300-12acf2c8fd1e, 3d4eae86-38df-4579-9da4-1e5f1514a9a5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:22:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Running pipeline stage MKMLDeleter
Shutdown handler de-registered
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b9b0943f-6861-4def-8037-0d58d0381920, 1d4eac60-90bc-4058-a796-063852e0c3f4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:22:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v2-1e5_v29
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v2-1e5-v29 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b99ba6ed-cee1-4edf-bc84-fb1c22e1302c, bd1061a8-5b2a-4747-b066-0d17f0d75b1b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:26:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
zonemercy-vingt-deux-v2-1e5_v29 status is now torndown due to DeploymentManager action