zonemercy-vingt-deux-v6-2e6

submission_id: zonemercy-vingt-deux-v6-2e6_v4
developer_uid: chai_backend_admin
best_of: 8
celo_rating: 1246.33
display_name: temp-8
family_friendly_score: 0.5714891212206838
family_friendly_standard_error: 0.008301649127257072
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': 'Bot: {message}\n', 'user_template': 'User: {message}\n', 'response_template': 'Bot:', 'truncate_by_message': False}
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: zonemercy/Vingt-Deux-v6-2e6
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: zonemercy/Vingt-Deux-v6-
model_name: temp-8
model_num_parameters: 22247282688.0
model_repo: zonemercy/Vingt-Deux-v6-2e6
model_size: 22B
num_battles: 3640
num_wins: 1803
ranking_group: single
status: torndown
submission_type: basic
timestamp: 2024-09-27T14:07:40+00:00
us_pacific_date: 2024-09-27
win_ratio: 0.49532967032967035
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-v6-2e6-v4-mkmlizer
Waiting for job on zonemercy-vingt-deux-v6-2e6-v4-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-gfv5v6-v1-mkmlizer
Shutdown handler not registered because Python interpreter is not running in the main thread
Waiting for job on zonemercy-vingt-deux-gfv5v6-v1-mkmlizer to finish
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer
Waiting for job on zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer to finish
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: Downloaded to shared memory in 104.692s
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp2mugzdaq, device:0
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: Downloaded to shared memory in 92.976s
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpbr_agc1r, device:0
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: Downloaded to shared memory in 106.259s
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpgck1_8i1, device:0
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: Downloaded to shared memory in 96.312s
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp3p41v2ns, device:0
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: quantized model in 45.161s
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: Processed model zonemercy/Vingt-Deux-v2-1e5 in 149.854s
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/config.json
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/tokenizer_config.json
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/special_tokens_map.json
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/tokenizer.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: quantized model in 47.425s
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: Processed model zonemercy/Vingt-Deux-v3-1e5v0 in 140.401s
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: quantized model in 46.853s
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: Processed model zonemercy/Vingt-Deux-v6-2e6 in 153.112s
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/flywheel_model.1.safetensors
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/special_tokens_map.json
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/config.json
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/tokenizer_config.json
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/tokenizer.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/config.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/special_tokens_map.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/tokenizer_config.json
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/tokenizer.json
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v2-1e5-v29/flywheel_model.0.safetensors
zonemercy-vingt-deux-v2-1e5-v29-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/507 [00:00<00:12, 39.79it/s]
Loading 0:   2%|▏         | 8/507 [00:00<00:15, 31.62it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:14, 33.48it/s]
Loading 0:   3%|▎         | 16/507 [00:00<00:15, 32.34it/s]
Loading 0:   4%|▍         | 21/507 [00:00<00:13, 35.49it/s]
Loading 0:   5%|▍         | 25/507 [00:00<00:14, 33.57it/s]
Loading 0:   6%|▌         | 29/507 [00:00<00:13, 34.67it/s]
Loading 0:   7%|▋         | 33/507 [00:01<00:15, 31.14it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:14, 33.02it/s]
Loading 0:   8%|▊         | 41/507 [00:01<00:15, 30.32it/s]
Loading 0:   9%|▉         | 47/507 [00:01<00:12, 35.45it/s]
Loading 0:  10%|█         | 51/507 [00:01<00:13, 34.01it/s]
Loading 0:  11%|█         | 55/507 [00:01<00:19, 23.64it/s]
Loading 0:  11%|█▏        | 58/507 [00:01<00:18, 24.67it/s]
Loading 0:  12%|█▏        | 63/507 [00:02<00:15, 28.30it/s]
Loading 0:  13%|█▎        | 67/507 [00:02<00:15, 28.74it/s]
Loading 0:  14%|█▍        | 71/507 [00:02<00:14, 30.92it/s]
Loading 0:  15%|█▍        | 75/507 [00:02<00:13, 32.25it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:13, 32.16it/s]
Loading 0:  17%|█▋        | 85/507 [00:02<00:11, 35.63it/s]
Loading 0:  18%|█▊        | 89/507 [00:02<00:13, 31.63it/s]
Loading 0:  19%|█▊        | 94/507 [00:02<00:11, 35.78it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:12, 31.90it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:11, 35.89it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:12, 31.98it/s]
Loading 0:  22%|██▏       | 112/507 [00:03<00:11, 35.71it/s]
Loading 0:  23%|██▎       | 116/507 [00:03<00:17, 23.00it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:15, 25.64it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:12, 29.72it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:13, 28.11it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:11, 32.42it/s]
Loading 0:  28%|██▊       | 140/507 [00:04<00:12, 30.56it/s]
Loading 0:  29%|██▊       | 145/507 [00:04<00:10, 34.90it/s]
Loading 0:  29%|██▉       | 149/507 [00:04<00:11, 31.92it/s]
Loading 0:  30%|███       | 154/507 [00:04<00:09, 35.97it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:10, 32.35it/s]
Loading 0:  32%|███▏      | 164/507 [00:05<00:09, 36.40it/s]
Loading 0:  33%|███▎      | 169/507 [00:05<00:12, 28.03it/s]
Loading 0:  34%|███▍      | 173/507 [00:05<00:11, 28.71it/s]
Loading 0:  35%|███▍      | 177/507 [00:05<00:11, 27.72it/s]
Loading 0:  36%|███▌      | 181/507 [00:05<00:10, 30.24it/s]
Loading 0:  36%|███▋      | 185/507 [00:05<00:11, 28.69it/s]
Loading 0:  37%|███▋      | 190/507 [00:06<00:09, 33.28it/s]
Loading 0:  38%|███▊      | 194/507 [00:06<00:10, 30.66it/s]
Loading 0:  39%|███▉      | 199/507 [00:06<00:08, 34.95it/s]
Loading 0:  40%|████      | 203/507 [00:06<00:09, 31.86it/s]
Loading 0:  41%|████▏     | 210/507 [00:06<00:07, 38.65it/s]
Loading 0:  42%|████▏     | 215/507 [00:06<00:07, 37.97it/s]
Loading 0:  43%|████▎     | 219/507 [00:06<00:09, 30.46it/s]
Loading 0:  44%|████▍     | 223/507 [00:07<00:08, 32.42it/s]
Loading 0:  45%|████▍     | 227/507 [00:07<00:10, 27.60it/s]
Loading 0:  46%|████▌     | 231/507 [00:07<00:10, 26.96it/s]
Loading 0:  46%|████▋     | 235/507 [00:07<00:09, 29.45it/s]
Loading 0:  47%|████▋     | 239/507 [00:07<00:10, 26.20it/s]
Loading 0:  48%|████▊     | 244/507 [00:07<00:08, 30.75it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:09, 28.39it/s]
Loading 0:  50%|████▉     | 253/507 [00:08<00:07, 32.47it/s]
Loading 0:  51%|█████     | 257/507 [00:08<00:08, 29.96it/s]
Loading 0:  52%|█████▏    | 262/507 [00:08<00:07, 34.50it/s]
Loading 0:  52%|█████▏    | 266/507 [00:08<00:07, 31.21it/s]
Loading 0:  53%|█████▎    | 271/507 [00:08<00:06, 34.97it/s]
Loading 0:  54%|█████▍    | 275/507 [00:08<00:07, 31.79it/s]
Loading 0:  55%|█████▌    | 280/507 [00:08<00:06, 35.88it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:08, 26.52it/s]
Loading 0:  57%|█████▋    | 288/507 [00:09<00:08, 26.15it/s]
Loading 0:  58%|█████▊    | 293/507 [00:09<00:07, 26.93it/s]
Loading 0:  59%|█████▉    | 298/507 [00:09<00:06, 31.43it/s]
Loading 0:  59%|█████▉    | 299/507 [00:20<00:06, 31.43it/s]
Loading 0:  59%|█████▉    | 300/507 [00:24<03:58,  1.15s/it]
Loading 0:  60%|█████▉    | 302/507 [00:24<03:16,  1.05it/s]
Loading 0:  61%|██████    | 307/507 [00:24<01:58,  1.69it/s]
Loading 0:  61%|██████    | 310/507 [00:24<01:29,  2.19it/s]
Loading 0:  62%|██████▏   | 314/507 [00:24<01:01,  3.16it/s]
Loading 0:  63%|██████▎   | 318/507 [00:24<00:42,  4.44it/s]
Loading 0:  64%|██████▎   | 322/507 [00:24<00:30,  6.06it/s]
Loading 0:  64%|██████▍   | 326/507 [00:24<00:22,  8.20it/s]
Loading 0:  65%|██████▌   | 330/507 [00:25<00:17, 10.33it/s]
Loading 0:  66%|██████▌   | 334/507 [00:25<00:12, 13.31it/s]
Loading 0:  67%|██████▋   | 339/507 [00:25<00:09, 17.41it/s]
Loading 0:  68%|██████▊   | 343/507 [00:25<00:10, 15.96it/s]
Loading 0:  68%|██████▊   | 347/507 [00:25<00:09, 17.74it/s]
Loading 0:  69%|██████▉   | 352/507 [00:25<00:06, 22.52it/s]
Loading 0:  70%|███████   | 356/507 [00:26<00:06, 23.45it/s]
Loading 0:  71%|███████   | 361/507 [00:26<00:05, 28.36it/s]
Loading 0:  72%|███████▏  | 365/507 [00:26<00:05, 27.88it/s]
Loading 0:  73%|███████▎  | 370/507 [00:26<00:04, 32.50it/s]
Loading 0:  74%|███████▍  | 374/507 [00:26<00:04, 30.58it/s]
Loading 0:  75%|███████▌  | 381/507 [00:26<00:03, 37.87it/s]
Loading 0:  76%|███████▌  | 386/507 [00:26<00:03, 37.40it/s]
Loading 0:  77%|███████▋  | 391/507 [00:27<00:03, 31.13it/s]
Loading 0:  78%|███████▊  | 395/507 [00:27<00:04, 24.69it/s]
Loading 0:  79%|███████▉  | 401/507 [00:27<00:03, 26.74it/s]
Loading 0:  80%|████████  | 406/507 [00:27<00:03, 30.87it/s]
Loading 0:  81%|████████  | 410/507 [00:27<00:03, 28.76it/s]
Loading 0:  82%|████████▏ | 415/507 [00:27<00:02, 32.74it/s]
Loading 0:  83%|████████▎ | 419/507 [00:28<00:02, 30.41it/s]
Loading 0:  84%|████████▎ | 424/507 [00:28<00:02, 34.64it/s]
Loading 0:  84%|████████▍ | 428/507 [00:28<00:02, 31.46it/s]
Loading 0:  85%|████████▌ | 433/507 [00:28<00:02, 35.08it/s]
Loading 0:  86%|████████▌ | 437/507 [00:28<00:02, 31.79it/s]
Loading 0:  88%|████████▊ | 444/507 [00:28<00:01, 38.57it/s]
Loading 0:  89%|████████▊ | 449/507 [00:28<00:01, 37.82it/s]
Loading 0:  90%|████████▉ | 454/507 [00:29<00:01, 34.88it/s]
Loading 0:  90%|█████████ | 458/507 [00:31<00:08,  6.11it/s]
Loading 0:  91%|█████████ | 461/507 [00:31<00:06,  7.17it/s]
Loading 0:  92%|█████████▏| 465/507 [00:31<00:04,  9.05it/s]
Loading 0:  93%|█████████▎| 470/507 [00:31<00:02, 12.47it/s]
Loading 0:  93%|█████████▎| 474/507 [00:31<00:02, 14.32it/s]
Loading 0:  94%|█████████▍| 479/507 [00:31<00:01, 18.62it/s]
Loading 0:  95%|█████████▌| 483/507 [00:32<00:01, 20.20it/s]
Loading 0:  96%|█████████▋| 488/507 [00:32<00:00, 25.01it/s]
Loading 0:  97%|█████████▋| 492/507 [00:32<00:00, 24.83it/s]
Loading 0:  98%|█████████▊| 497/507 [00:32<00:00, 29.39it/s]
Loading 0:  99%|█████████▉| 501/507 [00:32<00:00, 27.79it/s]
Loading 0: 100%|█████████▉| 506/507 [00:32<00:00, 32.42it/s]
                                                            
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/flywheel_model.1.safetensors
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v6-2e6-v4/flywheel_model.0.safetensors
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/flywheel_model.0.safetensors
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v3-1e5v0-v8/flywheel_model.1.safetensors
zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:22, 21.91it/s]
Loading 0:   2%|▏         | 10/507 [00:00<00:16, 30.48it/s]
Loading 0:   3%|▎         | 14/507 [00:00<00:18, 26.15it/s]
Loading 0:   4%|▎         | 19/507 [00:00<00:15, 30.68it/s]
Loading 0:   5%|▍         | 23/507 [00:00<00:17, 27.34it/s]
Loading 0:   6%|▌         | 28/507 [00:00<00:15, 31.19it/s]
Loading 0:   6%|▋         | 32/507 [00:01<00:17, 27.48it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:15, 30.81it/s]
Loading 0:   8%|▊         | 41/507 [00:01<00:16, 27.45it/s]
Loading 0:   9%|▉         | 46/507 [00:01<00:15, 30.58it/s]
Loading 0:  10%|▉         | 50/507 [00:01<00:16, 27.11it/s]
Loading 0:  10%|█         | 53/507 [00:02<00:24, 18.28it/s]
Loading 0:  11%|█         | 56/507 [00:02<00:25, 17.56it/s]
Loading 0:  12%|█▏        | 61/507 [00:02<00:20, 21.94it/s]
Loading 0:  13%|█▎        | 65/507 [00:02<00:20, 21.55it/s]
Loading 0:  14%|█▍        | 70/507 [00:02<00:17, 25.61it/s]
Loading 0:  14%|█▍        | 73/507 [00:02<00:16, 26.01it/s]
Loading 0:  16%|█▌        | 80/507 [00:03<00:14, 28.90it/s]
Loading 0:  17%|█▋        | 85/507 [00:03<00:13, 32.04it/s]
Loading 0:  18%|█▊        | 89/507 [00:03<00:14, 29.17it/s]
Loading 0:  19%|█▊        | 94/507 [00:03<00:12, 33.06it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:13, 29.77it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:11, 33.72it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:13, 30.55it/s]
Loading 0:  22%|██▏       | 112/507 [00:04<00:11, 34.15it/s]
Loading 0:  23%|██▎       | 116/507 [00:04<00:17, 22.12it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:15, 25.21it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:12, 29.52it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:13, 27.95it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:11, 31.88it/s]
Loading 0:  28%|██▊       | 140/507 [00:05<00:13, 27.93it/s]
Loading 0:  29%|██▊       | 145/507 [00:05<00:11, 31.53it/s]
Loading 0:  29%|██▉       | 149/507 [00:05<00:12, 29.30it/s]
Loading 0:  30%|███       | 154/507 [00:05<00:10, 32.57it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:11, 30.71it/s]
Loading 0:  32%|███▏      | 164/507 [00:05<00:09, 35.11it/s]
Loading 0:  33%|███▎      | 168/507 [00:05<00:09, 35.25it/s]
Loading 0:  34%|███▍      | 172/507 [00:06<00:13, 24.71it/s]
Loading 0:  35%|███▍      | 176/507 [00:06<00:13, 25.26it/s]
Loading 0:  36%|███▌      | 183/507 [00:06<00:09, 32.73it/s]
Loading 0:  37%|███▋      | 187/507 [00:06<00:09, 32.60it/s]
Loading 0:  38%|███▊      | 192/507 [00:06<00:08, 35.25it/s]
Loading 0:  39%|███▊      | 196/507 [00:06<00:09, 33.54it/s]
Loading 0:  40%|███▉      | 201/507 [00:06<00:08, 35.42it/s]
Loading 0:  40%|████      | 205/507 [00:07<00:08, 34.19it/s]
Loading 0:  41%|████▏     | 210/507 [00:07<00:08, 36.63it/s]
Loading 0:  42%|████▏     | 214/507 [00:07<00:08, 33.70it/s]
Loading 0:  43%|████▎     | 218/507 [00:07<00:08, 33.04it/s]
Loading 0:  44%|████▍     | 222/507 [00:07<00:08, 31.95it/s]
Loading 0:  45%|████▍     | 226/507 [00:07<00:12, 22.88it/s]
Loading 0:  45%|████▌     | 230/507 [00:08<00:12, 22.55it/s]
Loading 0:  46%|████▋     | 235/507 [00:08<00:10, 26.58it/s]
Loading 0:  47%|████▋     | 239/507 [00:08<00:10, 25.21it/s]
Loading 0:  48%|████▊     | 244/507 [00:08<00:09, 29.05it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:09, 26.98it/s]
Loading 0:  50%|████▉     | 253/507 [00:08<00:08, 31.07it/s]
Loading 0:  51%|█████     | 257/507 [00:09<00:08, 28.04it/s]
Loading 0:  52%|█████▏    | 262/507 [00:09<00:07, 31.70it/s]
Loading 0:  52%|█████▏    | 266/507 [00:09<00:08, 28.85it/s]
Loading 0:  53%|█████▎    | 271/507 [00:09<00:07, 32.11it/s]
Loading 0:  54%|█████▍    | 275/507 [00:09<00:07, 29.46it/s]
Loading 0:  55%|█████▌    | 280/507 [00:09<00:06, 32.91it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:09, 23.87it/s]
Loading 0:  57%|█████▋    | 287/507 [00:10<00:09, 22.04it/s]
Loading 0:  58%|█████▊    | 293/507 [00:10<00:08, 24.67it/s]
Loading 0:  59%|█████▉    | 298/507 [00:10<00:07, 28.44it/s]
Loading 0:  60%|█████▉    | 302/507 [00:25<03:34,  1.05s/it]
Loading 0:  61%|██████    | 307/507 [00:25<02:24,  1.39it/s]
Loading 0:  61%|██████▏   | 311/507 [00:26<01:46,  1.84it/s]
Loading 0:  63%|██████▎   | 318/507 [00:26<01:02,  3.02it/s]
Loading 0:  64%|██████▎   | 322/507 [00:26<00:47,  3.89it/s]
Loading 0:  64%|██████▍   | 327/507 [00:26<00:33,  5.40it/s]
Loading 0:  65%|██████▌   | 331/507 [00:26<00:25,  6.87it/s]
Loading 0:  66%|██████▌   | 335/507 [00:26<00:19,  8.76it/s]
Loading 0:  67%|██████▋   | 339/507 [00:26<00:15, 11.17it/s]
Loading 0:  68%|██████▊   | 343/507 [00:27<00:14, 11.40it/s]
Loading 0:  68%|██████▊   | 347/507 [00:27<00:11, 13.36it/s]
Loading 0:  69%|██████▉   | 352/507 [00:27<00:08, 17.62it/s]
Loading 0:  70%|███████   | 356/507 [00:27<00:07, 19.22it/s]
Loading 0:  71%|███████   | 361/507 [00:27<00:06, 23.80it/s]
Loading 0:  72%|███████▏  | 365/507 [00:27<00:05, 23.86it/s]
Loading 0:  73%|███████▎  | 370/507 [00:28<00:04, 28.12it/s]
Loading 0:  74%|███████▍  | 374/507 [00:28<00:04, 27.04it/s]
Loading 0:  75%|███████▍  | 379/507 [00:28<00:04, 31.35it/s]
Loading 0:  76%|███████▌  | 383/507 [00:28<00:04, 28.88it/s]
Loading 0:  77%|███████▋  | 388/507 [00:28<00:03, 33.10it/s]
Loading 0:  77%|███████▋  | 392/507 [00:28<00:03, 30.31it/s]
Loading 0:  78%|███████▊  | 396/507 [00:29<00:05, 21.88it/s]
Loading 0:  79%|███████▉  | 401/507 [00:29<00:04, 23.88it/s]
Loading 0:  80%|████████  | 406/507 [00:29<00:03, 26.95it/s]
Loading 0:  81%|████████  | 410/507 [00:29<00:03, 25.58it/s]
Loading 0:  82%|████████▏ | 415/507 [00:29<00:03, 28.81it/s]
Loading 0:  83%|████████▎ | 419/507 [00:29<00:03, 26.72it/s]
Loading 0:  84%|████████▎ | 424/507 [00:30<00:02, 30.29it/s]
Loading 0:  84%|████████▍ | 428/507 [00:30<00:02, 27.95it/s]
Loading 0:  85%|████████▌ | 433/507 [00:30<00:02, 31.48it/s]
Loading 0:  86%|████████▌ | 437/507 [00:30<00:02, 28.72it/s]
Loading 0:  87%|████████▋ | 442/507 [00:30<00:01, 32.62it/s]
Loading 0:  88%|████████▊ | 446/507 [00:30<00:02, 29.59it/s]
Loading 0:  89%|████████▉ | 451/507 [00:30<00:01, 33.52it/s]
Loading 0:  90%|████████▉ | 455/507 [00:33<00:09,  5.57it/s]
Loading 0:  91%|█████████ | 459/507 [00:33<00:06,  7.11it/s]
Loading 0:  92%|█████████▏| 465/507 [00:33<00:04, 10.08it/s]
Loading 0:  93%|█████████▎| 470/507 [00:33<00:02, 13.30it/s]
Loading 0:  93%|█████████▎| 474/507 [00:33<00:02, 14.97it/s]
Loading 0:  94%|█████████▍| 479/507 [00:33<00:01, 18.98it/s]
Loading 0:  95%|█████████▌| 483/507 [00:34<00:01, 20.16it/s]
Loading 0:  96%|█████████▋| 488/507 [00:34<00:00, 24.53it/s]
Loading 0:  97%|█████████▋| 492/507 [00:34<00:00, 24.15it/s]
Loading 0:  98%|█████████▊| 497/507 [00:34<00:00, 28.37it/s]
Loading 0:  99%|█████████▉| 501/507 [00:34<00:00, 26.89it/s]
Loading 0: 100%|█████████▉| 506/507 [00:34<00:00, 31.22it/s]
                                                            
zonemercy-vingt-deux-v6-2e6-v4-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:20, 24.71it/s]
Loading 0:   2%|▏         | 10/507 [00:00<00:15, 32.85it/s]
Loading 0:   3%|▎         | 14/507 [00:00<00:17, 28.75it/s]
Loading 0:   4%|▎         | 19/507 [00:00<00:14, 34.26it/s]
Loading 0:   5%|▍         | 23/507 [00:00<00:16, 30.12it/s]
Loading 0:   6%|▌         | 28/507 [00:00<00:13, 34.64it/s]
Loading 0:   6%|▋         | 32/507 [00:01<00:15, 30.33it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:13, 33.63it/s]
Loading 0:   8%|▊         | 41/507 [00:01<00:15, 30.07it/s]
Loading 0:   9%|▉         | 46/507 [00:01<00:13, 33.28it/s]
Loading 0:  10%|▉         | 50/507 [00:01<00:14, 30.58it/s]
Loading 0:  11%|█         | 54/507 [00:01<00:21, 21.16it/s]
Loading 0:  11%|█         | 57/507 [00:02<00:21, 21.31it/s]
Loading 0:  12%|█▏        | 61/507 [00:02<00:18, 24.30it/s]
Loading 0:  13%|█▎        | 65/507 [00:02<00:18, 24.08it/s]
Loading 0:  14%|█▍        | 70/507 [00:02<00:15, 28.54it/s]
Loading 0:  15%|█▍        | 75/507 [00:02<00:13, 31.37it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:13, 31.10it/s]
Loading 0:  17%|█▋        | 85/507 [00:02<00:12, 34.40it/s]
Loading 0:  18%|█▊        | 89/507 [00:03<00:13, 30.83it/s]
Loading 0:  19%|█▊        | 94/507 [00:03<00:12, 34.17it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:13, 30.15it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:12, 33.57it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:13, 30.18it/s]
Loading 0:  22%|██▏       | 112/507 [00:03<00:11, 33.20it/s]
Loading 0:  23%|██▎       | 116/507 [00:04<00:18, 21.25it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:15, 24.25it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:13, 28.35it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:14, 26.83it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:12, 30.74it/s]
Loading 0:  28%|██▊       | 140/507 [00:04<00:12, 28.41it/s]
Loading 0:  29%|██▊       | 145/507 [00:04<00:11, 32.03it/s]
Loading 0:  29%|██▉       | 149/507 [00:05<00:12, 29.35it/s]
Loading 0:  30%|███       | 154/507 [00:05<00:10, 32.98it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:11, 29.77it/s]
Loading 0:  32%|███▏      | 163/507 [00:05<00:10, 32.68it/s]
Loading 0:  33%|███▎      | 168/507 [00:05<00:10, 33.48it/s]
Loading 0:  34%|███▍      | 172/507 [00:06<00:14, 22.47it/s]
Loading 0:  35%|███▍      | 176/507 [00:06<00:14, 22.87it/s]
Loading 0:  36%|███▌      | 181/507 [00:06<00:11, 27.25it/s]
Loading 0:  36%|███▋      | 185/507 [00:06<00:12, 26.49it/s]
Loading 0:  37%|███▋      | 190/507 [00:06<00:10, 30.38it/s]
Loading 0:  38%|███▊      | 194/507 [00:06<00:11, 28.12it/s]
Loading 0:  39%|███▉      | 199/507 [00:06<00:09, 32.09it/s]
Loading 0:  40%|████      | 203/507 [00:07<00:10, 29.58it/s]
Loading 0:  41%|████      | 208/507 [00:07<00:08, 33.59it/s]
Loading 0:  42%|████▏     | 212/507 [00:07<00:09, 29.80it/s]
Loading 0:  43%|████▎     | 217/507 [00:07<00:08, 33.74it/s]
Loading 0:  44%|████▎     | 221/507 [00:07<00:09, 30.53it/s]
Loading 0:  44%|████▍     | 225/507 [00:07<00:12, 21.72it/s]
Loading 0:  45%|████▌     | 230/507 [00:08<00:11, 23.94it/s]
Loading 0:  46%|████▋     | 235/507 [00:08<00:09, 28.03it/s]
Loading 0:  47%|████▋     | 239/507 [00:08<00:10, 26.49it/s]
Loading 0:  48%|████▊     | 244/507 [00:08<00:08, 30.31it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:09, 27.61it/s]
Loading 0:  50%|████▉     | 253/507 [00:08<00:08, 31.24it/s]
Loading 0:  51%|█████     | 257/507 [00:08<00:08, 28.30it/s]
Loading 0:  52%|█████▏    | 262/507 [00:09<00:07, 32.27it/s]
Loading 0:  52%|█████▏    | 266/507 [00:09<00:08, 29.78it/s]
Loading 0:  53%|█████▎    | 271/507 [00:09<00:06, 33.74it/s]
Loading 0:  54%|█████▍    | 275/507 [00:09<00:07, 30.45it/s]
Loading 0:  55%|█████▌    | 280/507 [00:09<00:06, 34.60it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:08, 25.29it/s]
Loading 0:  57%|█████▋    | 288/507 [00:10<00:08, 25.28it/s]
Loading 0:  58%|█████▊    | 293/507 [00:10<00:08, 26.29it/s]
Loading 0:  59%|█████▉    | 298/507 [00:10<00:06, 30.54it/s]
Loading 0:  59%|█████▉    | 299/507 [00:25<00:06, 30.54it/s]
Loading 0:  59%|█████▉    | 300/507 [00:25<04:06,  1.19s/it]
Loading 0:  60%|█████▉    | 302/507 [00:25<03:22,  1.01it/s]
Loading 0:  61%|██████    | 307/507 [00:25<02:02,  1.63it/s]
Loading 0:  61%|██████    | 310/507 [00:25<01:32,  2.12it/s]
Loading 0:  62%|██████▏   | 313/507 [00:25<01:08,  2.81it/s]
Loading 0:  62%|██████▏   | 316/507 [00:25<00:51,  3.74it/s]
Loading 0:  63%|██████▎   | 320/507 [00:26<00:35,  5.30it/s]
Loading 0:  64%|██████▍   | 325/507 [00:26<00:22,  8.02it/s]
Loading 0:  65%|██████▍   | 329/507 [00:26<00:17, 10.12it/s]
Loading 0:  66%|██████▌   | 334/507 [00:26<00:12, 13.74it/s]
Loading 0:  67%|██████▋   | 339/507 [00:26<00:09, 17.58it/s]
Loading 0:  68%|██████▊   | 343/507 [00:26<00:10, 15.71it/s]
Loading 0:  68%|██████▊   | 347/507 [00:27<00:09, 17.31it/s]
Loading 0:  69%|██████▉   | 352/507 [00:27<00:07, 21.90it/s]
Loading 0:  70%|███████   | 356/507 [00:27<00:06, 22.74it/s]
Loading 0:  71%|███████   | 361/507 [00:27<00:05, 27.33it/s]
Loading 0:  72%|███████▏  | 365/507 [00:27<00:05, 26.80it/s]
Loading 0:  73%|███████▎  | 370/507 [00:27<00:04, 31.29it/s]
Loading 0:  74%|███████▍  | 374/507 [00:27<00:04, 29.73it/s]
Loading 0:  75%|███████▌  | 381/507 [00:28<00:03, 36.84it/s]
Loading 0:  76%|███████▌  | 386/507 [00:28<00:03, 36.13it/s]
Loading 0:  77%|███████▋  | 390/507 [00:28<00:04, 28.26it/s]
Loading 0:  78%|███████▊  | 394/507 [00:28<00:03, 29.86it/s]
Loading 0:  79%|███████▊  | 398/507 [00:28<00:04, 25.71it/s]
Loading 0:  79%|███████▉  | 401/507 [00:28<00:04, 23.46it/s]
Loading 0:  80%|████████  | 406/507 [00:29<00:03, 28.15it/s]
Loading 0:  81%|████████  | 410/507 [00:29<00:03, 26.57it/s]
Loading 0:  82%|████████▏ | 415/507 [00:29<00:03, 30.49it/s]
Loading 0:  83%|████████▎ | 419/507 [00:29<00:03, 28.08it/s]
Loading 0:  84%|████████▎ | 424/507 [00:29<00:02, 32.06it/s]
Loading 0:  84%|████████▍ | 428/507 [00:29<00:02, 29.51it/s]
Loading 0:  85%|████████▌ | 433/507 [00:29<00:02, 32.91it/s]
Loading 0:  86%|████████▌ | 437/507 [00:30<00:02, 30.15it/s]
Loading 0:  87%|████████▋ | 442/507 [00:30<00:01, 34.38it/s]
Loading 0:  88%|████████▊ | 446/507 [00:30<00:01, 30.95it/s]
Loading 0:  89%|████████▉ | 451/507 [00:30<00:01, 34.98it/s]
Loading 0:  90%|████████▉ | 455/507 [00:32<00:09,  5.66it/s]
Loading 0:  91%|█████████ | 459/507 [00:32<00:06,  7.21it/s]
Loading 0:  92%|█████████▏| 465/507 [00:33<00:04, 10.19it/s]
Loading 0:  93%|█████████▎| 470/507 [00:33<00:02, 13.35it/s]
Loading 0:  93%|█████████▎| 474/507 [00:33<00:02, 14.99it/s]
Loading 0:  94%|█████████▍| 479/507 [00:33<00:01, 18.97it/s]
Loading 0:  95%|█████████▌| 483/507 [00:33<00:01, 19.87it/s]
Loading 0:  96%|█████████▋| 488/507 [00:33<00:00, 24.14it/s]
Loading 0:  97%|█████████▋| 492/507 [00:33<00:00, 23.40it/s]
Loading 0:  98%|█████████▊| 497/507 [00:34<00:00, 25.25it/s]
Loading 0:  99%|█████████▉| 501/507 [00:34<00:00, 24.79it/s]
Loading 0: 100%|█████████▉| 506/507 [00:34<00:00, 29.21it/s]
                                                            
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: quantized model in 47.773s
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: Processed model zonemercy/Vingt-Deux-v1-1e5 in 144.086s
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/config.json
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/special_tokens_map.json
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/tokenizer_config.json
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/tokenizer.json
Job zonemercy-vingt-deux-v2-1e5-v29-mkmlizer completed after 218.2s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v2-1e5-v29-mkmlizer
Pipeline stage MKMLizer completed in 218.88s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.25s
run pipeline stage %s
Job zonemercy-vingt-deux-v6-2e6-v4-mkmlizer completed after 218.38s with status: succeeded
Running pipeline stage MKMLDeployer
Stopping job with name zonemercy-vingt-deux-v6-2e6-v4-mkmlizer
Creating inference service zonemercy-vingt-deux-v2-1e5-v29
Pipeline stage MKMLizer completed in 219.23s
run pipeline stage %s
Waiting for inference service zonemercy-vingt-deux-v2-1e5-v29 to be ready
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.27s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-v6-2e6-v4
Job zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer completed after 218.89s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v3-1e5v0-v8-mkmlizer
Waiting for inference service zonemercy-vingt-deux-v6-2e6-v4 to be ready
Pipeline stage MKMLizer completed in 219.92s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.28s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-v3-1e5v0-v8
Waiting for inference service zonemercy-vingt-deux-v3-1e5v0-v8 to be ready
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/flywheel_model.1.safetensors
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v28/flywheel_model.0.safetensors
zonemercy-vingt-deux-v1-1e5-v28-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 5/507 [00:00<00:22, 22.72it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:13, 37.78it/s]
Loading 0:   3%|▎         | 17/507 [00:00<00:13, 37.49it/s]
Loading 0:   4%|▍         | 22/507 [00:00<00:12, 38.44it/s]
Loading 0:   5%|▌         | 27/507 [00:00<00:11, 40.42it/s]
Loading 0:   6%|▋         | 32/507 [00:00<00:14, 32.58it/s]
Loading 0:   8%|▊         | 39/507 [00:01<00:11, 39.08it/s]
Loading 0:   9%|▊         | 44/507 [00:01<00:12, 37.88it/s]
Loading 0:  10%|▉         | 49/507 [00:01<00:15, 30.29it/s]
Loading 0:  10%|█         | 53/507 [00:01<00:20, 22.51it/s]
Loading 0:  11%|█         | 56/507 [00:01<00:21, 21.13it/s]
Loading 0:  12%|█▏        | 61/507 [00:02<00:17, 25.71it/s]
Loading 0:  13%|█▎        | 65/507 [00:02<00:18, 24.45it/s]
Loading 0:  14%|█▍        | 70/507 [00:02<00:15, 28.76it/s]
Loading 0:  15%|█▍        | 75/507 [00:02<00:13, 31.09it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:14, 30.07it/s]
Loading 0:  17%|█▋        | 85/507 [00:02<00:12, 33.08it/s]
Loading 0:  18%|█▊        | 89/507 [00:02<00:14, 28.78it/s]
Loading 0:  19%|█▊        | 94/507 [00:03<00:12, 32.04it/s]
Loading 0:  19%|█▉        | 98/507 [00:03<00:14, 28.52it/s]
Loading 0:  20%|██        | 103/507 [00:03<00:12, 31.76it/s]
Loading 0:  21%|██        | 107/507 [00:03<00:14, 28.54it/s]
Loading 0:  22%|██▏       | 112/507 [00:03<00:12, 31.70it/s]
Loading 0:  23%|██▎       | 116/507 [00:04<00:19, 20.39it/s]
Loading 0:  24%|██▍       | 122/507 [00:04<00:16, 23.31it/s]
Loading 0:  25%|██▌       | 127/507 [00:04<00:13, 27.34it/s]
Loading 0:  26%|██▌       | 131/507 [00:04<00:14, 25.33it/s]
Loading 0:  27%|██▋       | 136/507 [00:04<00:12, 29.37it/s]
Loading 0:  28%|██▊       | 140/507 [00:04<00:13, 27.36it/s]
Loading 0:  29%|██▊       | 145/507 [00:04<00:11, 31.22it/s]
Loading 0:  29%|██▉       | 149/507 [00:05<00:12, 28.28it/s]
Loading 0:  30%|███       | 154/507 [00:05<00:11, 31.92it/s]
Loading 0:  31%|███       | 158/507 [00:05<00:12, 28.76it/s]
Loading 0:  32%|███▏      | 163/507 [00:05<00:10, 32.85it/s]
Loading 0:  33%|███▎      | 168/507 [00:05<00:10, 33.90it/s]
Loading 0:  34%|███▍      | 172/507 [00:06<00:14, 22.35it/s]
Loading 0:  35%|███▍      | 176/507 [00:06<00:14, 22.50it/s]
Loading 0:  36%|███▌      | 181/507 [00:06<00:12, 26.91it/s]
Loading 0:  36%|███▋      | 185/507 [00:06<00:12, 25.83it/s]
Loading 0:  37%|███▋      | 190/507 [00:06<00:10, 30.07it/s]
Loading 0:  38%|███▊      | 194/507 [00:06<00:11, 27.82it/s]
Loading 0:  39%|███▉      | 199/507 [00:06<00:09, 31.57it/s]
Loading 0:  40%|████      | 203/507 [00:07<00:10, 28.65it/s]
Loading 0:  41%|████      | 208/507 [00:07<00:09, 33.11it/s]
Loading 0:  42%|████▏     | 212/507 [00:07<00:10, 29.43it/s]
Loading 0:  43%|████▎     | 217/507 [00:07<00:08, 33.51it/s]
Loading 0:  44%|████▎     | 221/507 [00:07<00:09, 30.37it/s]
Loading 0:  44%|████▍     | 225/507 [00:07<00:13, 21.68it/s]
Loading 0:  45%|████▌     | 230/507 [00:08<00:11, 23.50it/s]
Loading 0:  46%|████▋     | 235/507 [00:08<00:09, 27.52it/s]
Loading 0:  47%|████▋     | 239/507 [00:08<00:10, 25.62it/s]
Loading 0:  48%|████▊     | 244/507 [00:08<00:08, 29.62it/s]
Loading 0:  49%|████▉     | 248/507 [00:08<00:09, 27.05it/s]
Loading 0:  50%|████▉     | 253/507 [00:08<00:08, 31.04it/s]
Loading 0:  51%|█████     | 257/507 [00:08<00:08, 28.10it/s]
Loading 0:  52%|█████▏    | 262/507 [00:09<00:07, 31.99it/s]
Loading 0:  52%|█████▏    | 266/507 [00:09<00:08, 28.52it/s]
Loading 0:  53%|█████▎    | 271/507 [00:09<00:07, 31.93it/s]
Loading 0:  54%|█████▍    | 275/507 [00:09<00:08, 28.80it/s]
Loading 0:  55%|█████▌    | 280/507 [00:09<00:07, 32.41it/s]
Loading 0:  56%|█████▌    | 284/507 [00:09<00:09, 24.15it/s]
Loading 0:  57%|█████▋    | 287/507 [00:10<00:10, 21.88it/s]
Loading 0:  58%|█████▊    | 293/507 [00:10<00:08, 24.71it/s]
Loading 0:  59%|█████▉    | 298/507 [00:10<00:07, 28.78it/s]
Loading 0:  60%|█████▉    | 302/507 [00:25<03:34,  1.05s/it]
Loading 0:  61%|██████    | 307/507 [00:25<02:23,  1.39it/s]
Loading 0:  61%|██████▏   | 311/507 [00:26<01:46,  1.84it/s]
Loading 0:  62%|██████▏   | 316/507 [00:26<01:11,  2.68it/s]
Loading 0:  63%|██████▎   | 320/507 [00:26<00:53,  3.53it/s]
Loading 0:  64%|██████▍   | 327/507 [00:26<00:31,  5.67it/s]
Loading 0:  65%|██████▌   | 331/507 [00:26<00:24,  7.06it/s]
Loading 0:  66%|██████▌   | 335/507 [00:26<00:19,  8.86it/s]
Loading 0:  67%|██████▋   | 339/507 [00:26<00:14, 11.21it/s]
Loading 0:  68%|██████▊   | 343/507 [00:27<00:14, 11.47it/s]
Loading 0:  68%|██████▊   | 347/507 [00:27<00:12, 13.30it/s]
Loading 0:  69%|██████▉   | 352/507 [00:27<00:08, 17.46it/s]
Loading 0:  70%|███████   | 356/507 [00:27<00:08, 18.69it/s]
Loading 0:  71%|███████   | 361/507 [00:27<00:06, 23.16it/s]
Loading 0:  72%|███████▏  | 365/507 [00:28<00:06, 23.05it/s]
Loading 0:  73%|███████▎  | 370/507 [00:28<00:05, 27.33it/s]
Loading 0:  74%|███████▍  | 374/507 [00:28<00:05, 26.04it/s]
Loading 0:  75%|███████▍  | 379/507 [00:28<00:04, 30.42it/s]
Loading 0:  76%|███████▌  | 383/507 [00:28<00:04, 27.95it/s]
Loading 0:  77%|███████▋  | 388/507 [00:28<00:03, 32.28it/s]
Loading 0:  77%|███████▋  | 392/507 [00:28<00:03, 29.62it/s]
Loading 0:  78%|███████▊  | 396/507 [00:29<00:05, 21.89it/s]
Loading 0:  79%|███████▉  | 401/507 [00:29<00:04, 23.61it/s]
Loading 0:  80%|████████  | 406/507 [00:29<00:03, 27.65it/s]
Loading 0:  81%|████████  | 410/507 [00:29<00:03, 25.75it/s]
Loading 0:  82%|████████▏ | 415/507 [00:29<00:03, 29.57it/s]
Loading 0:  83%|████████▎ | 419/507 [00:29<00:03, 27.16it/s]
Loading 0:  84%|████████▎ | 424/507 [00:30<00:02, 30.31it/s]
Loading 0:  84%|████████▍ | 428/507 [00:30<00:02, 27.45it/s]
Loading 0:  85%|████████▌ | 433/507 [00:30<00:02, 31.15it/s]
Loading 0:  86%|████████▌ | 437/507 [00:30<00:02, 28.22it/s]
Loading 0:  87%|████████▋ | 442/507 [00:30<00:02, 32.29it/s]
Loading 0:  88%|████████▊ | 446/507 [00:30<00:02, 29.06it/s]
Loading 0:  89%|████████▉ | 451/507 [00:30<00:01, 33.07it/s]
Loading 0:  90%|████████▉ | 455/507 [00:33<00:09,  5.49it/s]
Loading 0:  91%|█████████ | 459/507 [00:33<00:06,  6.97it/s]
Loading 0:  92%|█████████▏| 465/507 [00:33<00:04,  9.85it/s]
Loading 0:  93%|█████████▎| 472/507 [00:33<00:02, 14.33it/s]
Loading 0:  94%|█████████▍| 476/507 [00:33<00:01, 16.33it/s]
Loading 0:  95%|█████████▍| 481/507 [00:34<00:01, 19.90it/s]
Loading 0:  96%|█████████▌| 485/507 [00:34<00:01, 21.68it/s]
Loading 0:  97%|█████████▋| 490/507 [00:34<00:00, 24.89it/s]
Loading 0:  97%|█████████▋| 494/507 [00:34<00:00, 25.06it/s]
Loading 0:  98%|█████████▊| 498/507 [00:34<00:00, 27.72it/s]
Loading 0:  99%|█████████▉| 502/507 [00:34<00:00, 26.15it/s]
Loading 0: 100%|█████████▉| 506/507 [00:34<00:00, 27.60it/s]
                                                            
Job zonemercy-vingt-deux-v1-1e5-v28-mkmlizer completed after 239.14s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v1-1e5-v28-mkmlizer
Pipeline stage MKMLizer completed in 239.79s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.31s
run pipeline stage %s
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: Downloaded to shared memory in 207.378s
Running pipeline stage MKMLDeployer
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp5lk5x204, device:0
Creating inference service zonemercy-vingt-deux-v1-1e5-v28
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Waiting for inference service zonemercy-vingt-deux-v1-1e5-v28 to be ready
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: quantized model in 52.803s
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: Processed model zonemercy/Vingt-Deux-gfv5v6 in 260.181s
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/special_tokens_map.json
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/config.json
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/tokenizer_config.json
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/tokenizer.json
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/flywheel_model.1.safetensors
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-gfv5v6-v1/flywheel_model.0.safetensors
zonemercy-vingt-deux-gfv5v6-v1-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/507 [00:00<00:14, 34.01it/s]
Loading 0:   2%|▏         | 8/507 [00:00<00:17, 27.86it/s]
Loading 0:   2%|▏         | 12/507 [00:00<00:18, 27.15it/s]
Loading 0:   3%|▎         | 15/507 [00:00<00:21, 22.96it/s]
Loading 0:   4%|▎         | 19/507 [00:00<00:18, 26.61it/s]
Loading 0:   5%|▍         | 23/507 [00:00<00:16, 29.35it/s]
Loading 0:   5%|▌         | 27/507 [00:01<00:27, 17.76it/s]
Loading 0:   6%|▌         | 31/507 [00:01<00:22, 21.08it/s]
Loading 0:   7%|▋         | 34/507 [00:01<00:21, 21.62it/s]
Loading 0:   7%|▋         | 37/507 [00:01<00:20, 23.19it/s]
Loading 0:   8%|▊         | 40/507 [00:01<00:19, 23.61it/s]
Loading 0:   8%|▊         | 43/507 [00:01<00:19, 23.89it/s]
Loading 0:   9%|▉         | 46/507 [00:01<00:18, 25.00it/s]
Loading 0:  10%|▉         | 50/507 [00:02<00:16, 28.01it/s]
Loading 0:  10%|█         | 53/507 [00:02<00:23, 19.65it/s]
Loading 0:  11%|█         | 56/507 [00:02<00:21, 20.99it/s]
Loading 0:  12%|█▏        | 59/507 [00:02<00:25, 17.44it/s]
Loading 0:  13%|█▎        | 64/507 [00:02<00:19, 23.12it/s]
Loading 0:  13%|█▎        | 67/507 [00:02<00:18, 23.43it/s]
Loading 0:  14%|█▍        | 70/507 [00:03<00:18, 23.66it/s]
Loading 0:  14%|█▍        | 73/507 [00:03<00:17, 24.85it/s]
Loading 0:  15%|█▌        | 77/507 [00:03<00:15, 27.44it/s]
Loading 0:  16%|█▌        | 80/507 [00:03<00:21, 19.42it/s]
Loading 0:  16%|█▋        | 83/507 [00:03<00:22, 18.82it/s]
Loading 0:  17%|█▋        | 87/507 [00:03<00:18, 22.53it/s]
Loading 0:  18%|█▊        | 91/507 [00:03<00:16, 25.84it/s]
Loading 0:  19%|█▊        | 94/507 [00:04<00:16, 24.79it/s]
Loading 0:  19%|█▉        | 97/507 [00:04<00:16, 25.16it/s]
Loading 0:  20%|█▉        | 100/507 [00:04<00:20, 19.92it/s]
Loading 0:  21%|██        | 105/507 [00:04<00:15, 25.38it/s]
Loading 0:  21%|██▏       | 108/507 [00:04<00:22, 17.43it/s]
Loading 0:  22%|██▏       | 112/507 [00:04<00:19, 20.74it/s]
Loading 0:  23%|██▎       | 115/507 [00:05<00:18, 21.45it/s]
Loading 0:  23%|██▎       | 118/507 [00:05<00:16, 23.04it/s]
Loading 0:  24%|██▍       | 121/507 [00:05<00:16, 23.50it/s]
Loading 0:  24%|██▍       | 124/507 [00:05<00:16, 22.71it/s]
Loading 0:  25%|██▌       | 127/507 [00:05<00:15, 24.35it/s]
Loading 0:  26%|██▌       | 131/507 [00:05<00:13, 27.12it/s]
Loading 0:  26%|██▋       | 134/507 [00:05<00:19, 18.98it/s]
Loading 0:  27%|██▋       | 137/507 [00:06<00:18, 19.74it/s]
Loading 0:  28%|██▊       | 140/507 [00:06<00:21, 17.04it/s]
Loading 0:  29%|██▊       | 145/507 [00:06<00:16, 22.61it/s]
Loading 0:  29%|██▉       | 148/507 [00:06<00:16, 22.28it/s]
Loading 0:  30%|██▉       | 151/507 [00:06<00:15, 22.79it/s]
Loading 0:  30%|███       | 154/507 [00:06<00:14, 24.17it/s]
Loading 0:  31%|███       | 158/507 [00:06<00:12, 27.09it/s]
Loading 0:  32%|███▏      | 161/507 [00:07<00:17, 19.90it/s]
Loading 0:  32%|███▏      | 164/507 [00:07<00:16, 21.11it/s]
Loading 0:  33%|███▎      | 167/507 [00:07<00:19, 17.76it/s]
Loading 0:  34%|███▍      | 172/507 [00:07<00:14, 23.31it/s]
Loading 0:  35%|███▍      | 175/507 [00:07<00:14, 23.01it/s]
Loading 0:  35%|███▌      | 178/507 [00:07<00:14, 23.21it/s]
Loading 0:  36%|███▌      | 181/507 [00:08<00:13, 24.67it/s]
Loading 0:  36%|███▋      | 185/507 [00:08<00:11, 27.91it/s]
Loading 0:  37%|███▋      | 188/507 [00:08<00:16, 19.28it/s]
Loading 0:  38%|███▊      | 191/507 [00:08<00:15, 20.50it/s]
Loading 0:  38%|███▊      | 194/507 [00:08<00:18, 17.38it/s]
Loading 0:  39%|███▉      | 199/507 [00:08<00:13, 22.86it/s]
Loading 0:  40%|███▉      | 202/507 [00:09<00:12, 23.51it/s]
Loading 0:  40%|████      | 205/507 [00:09<00:12, 23.76it/s]
Loading 0:  41%|████      | 208/507 [00:09<00:12, 24.62it/s]
Loading 0:  42%|████▏     | 212/507 [00:09<00:10, 27.26it/s]
Loading 0:  42%|████▏     | 215/507 [00:09<00:15, 18.97it/s]
Loading 0:  43%|████▎     | 218/507 [00:09<00:14, 20.26it/s]
Loading 0:  44%|████▎     | 221/507 [00:10<00:16, 17.34it/s]
Loading 0:  45%|████▍     | 226/507 [00:10<00:12, 22.85it/s]
Loading 0:  45%|████▌     | 229/507 [00:10<00:11, 23.29it/s]
Loading 0:  46%|████▌     | 232/507 [00:10<00:11, 23.15it/s]
Loading 0:  46%|████▋     | 235/507 [00:10<00:11, 24.42it/s]
Loading 0:  47%|████▋     | 239/507 [00:10<00:09, 27.29it/s]
Loading 0:  48%|████▊     | 242/507 [00:10<00:13, 19.01it/s]
Loading 0:  48%|████▊     | 245/507 [00:11<00:12, 20.39it/s]
Loading 0:  49%|████▉     | 248/507 [00:11<00:14, 17.60it/s]
Loading 0:  50%|████▉     | 253/507 [00:11<00:11, 22.88it/s]
Loading 0:  50%|█████     | 256/507 [00:11<00:10, 23.35it/s]
Loading 0:  51%|█████     | 259/507 [00:11<00:10, 23.63it/s]
Loading 0:  52%|█████▏    | 262/507 [00:11<00:09, 24.93it/s]
Loading 0:  52%|█████▏    | 266/507 [00:11<00:08, 27.94it/s]
Loading 0:  53%|█████▎    | 269/507 [00:12<00:11, 19.87it/s]
Loading 0:  54%|█████▎    | 272/507 [00:12<00:11, 20.97it/s]
Loading 0:  54%|█████▍    | 275/507 [00:12<00:13, 17.69it/s]
Loading 0:  55%|█████▌    | 280/507 [00:12<00:09, 23.32it/s]
Loading 0:  56%|█████▌    | 283/507 [00:12<00:09, 23.95it/s]
Loading 0:  56%|█████▋    | 286/507 [00:12<00:09, 24.15it/s]
Loading 0:  57%|█████▋    | 289/507 [00:12<00:08, 25.50it/s]
Loading 0:  58%|█████▊    | 293/507 [00:13<00:07, 28.23it/s]
Loading 0:  58%|█████▊    | 296/507 [00:13<00:10, 20.29it/s]
Loading 0:  59%|█████▉    | 299/507 [00:13<00:09, 21.40it/s]
Loading 0:  60%|█████▉    | 302/507 [00:27<04:50,  1.42s/it]
Loading 0:  60%|██████    | 305/507 [00:28<03:25,  1.02s/it]
Loading 0:  61%|██████    | 308/507 [00:28<02:26,  1.36it/s]
Loading 0:  61%|██████▏   | 311/507 [00:28<01:46,  1.84it/s]
Loading 0:  62%|██████▏   | 316/507 [00:28<01:02,  3.04it/s]
Loading 0:  63%|██████▎   | 320/507 [00:28<00:43,  4.30it/s]
Loading 0:  64%|██████▎   | 323/507 [00:28<00:36,  5.09it/s]
Loading 0:  64%|██████▍   | 326/507 [00:29<00:28,  6.46it/s]
Loading 0:  65%|██████▍   | 329/507 [00:29<00:23,  7.48it/s]
Loading 0:  66%|██████▌   | 334/507 [00:29<00:15, 11.08it/s]
Loading 0:  66%|██████▋   | 337/507 [00:29<00:13, 12.86it/s]
Loading 0:  67%|██████▋   | 340/507 [00:29<00:11, 14.61it/s]
Loading 0:  68%|██████▊   | 343/507 [00:29<00:09, 16.96it/s]
Loading 0:  68%|██████▊   | 347/507 [00:29<00:07, 20.58it/s]
Loading 0:  69%|██████▉   | 350/507 [00:30<00:09, 16.07it/s]
Loading 0:  70%|██████▉   | 353/507 [00:30<00:08, 17.80it/s]
Loading 0:  70%|███████   | 356/507 [00:30<00:09, 15.82it/s]
Loading 0:  71%|███████   | 361/507 [00:30<00:06, 21.09it/s]
Loading 0:  72%|███████▏  | 364/507 [00:30<00:06, 22.05it/s]
Loading 0:  72%|███████▏  | 367/507 [00:30<00:06, 22.53it/s]
Loading 0:  73%|███████▎  | 370/507 [00:31<00:05, 23.97it/s]
Loading 0:  74%|███████▍  | 374/507 [00:31<00:04, 27.12it/s]
Loading 0:  74%|███████▍  | 377/507 [00:31<00:06, 19.22it/s]
Loading 0:  75%|███████▍  | 380/507 [00:31<00:06, 20.37it/s]
Loading 0:  76%|███████▌  | 383/507 [00:31<00:07, 16.88it/s]
Loading 0:  77%|███████▋  | 388/507 [00:31<00:05, 22.21it/s]
Loading 0:  77%|███████▋  | 391/507 [00:32<00:05, 22.61it/s]
Loading 0:  78%|███████▊  | 394/507 [00:32<00:04, 22.82it/s]
Loading 0:  78%|███████▊  | 397/507 [00:32<00:04, 24.35it/s]
Loading 0:  79%|███████▉  | 401/507 [00:32<00:03, 27.40it/s]
Loading 0:  80%|███████▉  | 404/507 [00:32<00:05, 18.62it/s]
Loading 0:  80%|████████  | 407/507 [00:32<00:05, 19.95it/s]
Loading 0:  81%|████████  | 410/507 [00:33<00:05, 16.69it/s]
Loading 0:  82%|████████▏ | 415/507 [00:33<00:04, 21.94it/s]
Loading 0:  82%|████████▏ | 418/507 [00:33<00:03, 22.41it/s]
Loading 0:  83%|████████▎ | 421/507 [00:33<00:03, 22.70it/s]
Loading 0:  84%|████████▎ | 424/507 [00:33<00:03, 23.45it/s]
Loading 0:  84%|████████▍ | 428/507 [00:33<00:03, 26.32it/s]
Loading 0:  85%|████████▌ | 431/507 [00:34<00:04, 17.88it/s]
Loading 0:  86%|████████▌ | 434/507 [00:34<00:03, 19.48it/s]
Loading 0:  86%|████████▌ | 437/507 [00:34<00:04, 16.83it/s]
Loading 0:  87%|████████▋ | 442/507 [00:34<00:02, 21.93it/s]
Loading 0:  88%|████████▊ | 445/507 [00:34<00:02, 22.44it/s]
Loading 0:  88%|████████▊ | 448/507 [00:34<00:02, 22.81it/s]
Loading 0:  89%|████████▉ | 451/507 [00:34<00:02, 24.42it/s]
Loading 0:  90%|████████▉ | 455/507 [00:34<00:01, 27.36it/s]
Loading 0:  90%|█████████ | 458/507 [00:35<00:02, 18.88it/s]
Loading 0:  91%|█████████ | 461/507 [00:35<00:02, 20.44it/s]
Loading 0:  92%|█████████▏| 464/507 [00:35<00:02, 16.79it/s]
Loading 0:  93%|█████████▎| 469/507 [00:35<00:01, 22.05it/s]
Loading 0:  93%|█████████▎| 472/507 [00:35<00:01, 22.55it/s]
Loading 0:  94%|█████████▎| 475/507 [00:35<00:01, 22.95it/s]
Loading 0:  94%|█████████▍| 478/507 [00:36<00:01, 24.47it/s]
Loading 0:  95%|█████████▌| 482/507 [00:36<00:00, 26.99it/s]
Loading 0:  96%|█████████▌| 485/507 [00:38<00:05,  4.28it/s]
Loading 0:  96%|█████████▋| 488/507 [00:38<00:03,  5.43it/s]
Loading 0:  97%|█████████▋| 491/507 [00:38<00:02,  7.02it/s]
Loading 0:  97%|█████████▋| 494/507 [00:38<00:01,  8.82it/s]
Loading 0:  98%|█████████▊| 497/507 [00:39<00:00, 10.99it/s]
Loading 0:  99%|█████████▊| 500/507 [00:39<00:00, 13.14it/s]
Loading 0:  99%|█████████▉| 503/507 [00:39<00:00, 14.93it/s]
Loading 0: 100%|█████████▉| 506/507 [00:39<00:00, 17.35it/s]
                                                            
Job zonemercy-vingt-deux-gfv5v6-v1-mkmlizer completed after 300.77s with status: succeeded
Stopping job with name zonemercy-vingt-deux-gfv5v6-v1-mkmlizer
Pipeline stage MKMLizer completed in 301.84s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.29s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-gfv5v6-v1
Waiting for inference service zonemercy-vingt-deux-gfv5v6-v1 to be ready
Inference service zonemercy-vingt-deux-v6-2e6-v4 ready after 230.50821113586426s
Pipeline stage MKMLDeployer completed in 231.67s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.1409964561462402s
Received healthy response to inference request in 2.4466605186462402s
Inference service zonemercy-vingt-deux-v2-1e5-v29 ready after 240.5290093421936s
Pipeline stage MKMLDeployer completed in 241.72s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.5495691299438477s
Inference service zonemercy-vingt-deux-v3-1e5v0-v8 ready after 240.5324902534485s
Pipeline stage MKMLDeployer completed in 241.62s
Received healthy response to inference request in 2.3935165405273438s
Received healthy response to inference request in 2.8595969676971436s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.465632200241089s
Received healthy response to inference request in 2.4500367641448975s
5 requests
0 failed requests
5th percentile: 2.404145336151123
Received healthy response to inference request in 2.8449018001556396s
10th percentile: 2.4147741317749025
20th percentile: 2.436031723022461
30th percentile: 2.45045485496521
40th percentile: 2.4580435276031496
50th percentile: 2.465632200241089
60th percentile: 2.4992069721221926
70th percentile: 2.532781744003296
Received healthy response to inference request in 2.3773341178894043s
80th percentile: 2.6678545951843264
90th percentile: 2.904425525665283
95th percentile: 3.0227109909057615
99th percentile: 3.1173393630981443
Received healthy response to inference request in 2.6949636936187744s
mean time: 2.5992749691009522
Pipeline stage StressChecker completed in 18.74s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.3296053409576416s
Received healthy response to inference request in 4.669405221939087s
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.94s
Shutdown handler de-registered
zonemercy-vingt-deux-v6-2e6_v4 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 2.5488362312316895s
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.28s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.20s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service zonemercy-vingt-deux-v6-2e6-v4-profiler
Waiting for inference service zonemercy-vingt-deux-v6-2e6-v4-profiler to be ready
Tearing down inference service zonemercy-vingt-deux-v6-2e6-v4-profiler
%s, retrying in %s seconds...
Creating inference service zonemercy-vingt-deux-v6-2e6-v4-profiler
Waiting for inference service zonemercy-vingt-deux-v6-2e6-v4-profiler to be ready
Tearing down inference service zonemercy-vingt-deux-v6-2e6-v4-profiler
%s, retrying in %s seconds...
Creating inference service zonemercy-vingt-deux-v6-2e6-v4-profiler
Ignoring service zonemercy-vingt-deux-v6-2e6-v4-profiler already deployed
Waiting for inference service zonemercy-vingt-deux-v6-2e6-v4-profiler to be ready
Tearing down inference service zonemercy-vingt-deux-v6-2e6-v4-profiler
clean up pipeline due to error=%s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.23s
Shutdown handler de-registered
zonemercy-vingt-deux-v6-2e6_v4 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ed5c26c8-fe20-4434-8c36-bf93a48b15d8, 4385c3ac-f3d6-403d-a486-3aba014e1ef4\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:16:41 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'54758639-5113-4a55-b39b-430aaf6e7f04, 412c9893-d978-46ed-8441-33c11876f3d7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:21:27 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1c127999-df58-4ec4-b389-e2f01f34d76c, d12c9d8b-bd53-4922-9b0b-1cec708b4cff\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:26:31 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'25d04321-558e-4049-95af-42a6e5512898, 68713ea8-62f5-4c72-840b-8092ad0eedae\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:31:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'232708f2-aa05-43e8-967b-ec777c0955d9, a6701e24-91c8-4c09-9f00-774340e4c9fe\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:38:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5a41ecfc-b174-4da7-b31d-e3511c14a990, 6b7ac1ca-c043-45a8-adac-d4520103017b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:41:33 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'15df66ae-e91f-4df0-a80b-d6841ba940e0, be8bd0c4-7d87-46af-b00c-3c247e9168a7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:46:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b4f16cce-8011-4bf5-83d7-632e1595dbfd, 87232128-bf3e-429c-b17d-78f5da64092a\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:51:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-viral-ss-v0-1e5_v5
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'34baf328-5d9a-4fbe-9a89-6321db14aa5e, 300053af-4ac5-4f20-9c76-085365a8b742\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:56:40 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ab69ab31-1f68-401d-b638-3e00a86aa542, 4cb63e4b-dd24-431d-9444-4280c945c36e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:01:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'410a4b25-17b7-435d-819f-5690f25a31ed, 69746889-af67-4943-8d45-0c3e0de7f58e\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:06:48 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'eb48043f-2b72-40c8-8611-9a87e8beee6d, 1db8d249-b289-4ba8-8e53-06f2e4a5e90d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:11:48 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'68ec83b6-15e4-41b6-83fc-b9485c2ec561, 8d03fc80-f2dd-4410-bcdc-81329c5e4576\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:16:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8ee485a2-ba9d-4122-add0-2bd1d2559617, 623bda72-aab2-4b3f-93b6-a51aac55e566\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:21:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'886492e8-56a7-48e5-8d69-f5a4b56aa8a2, c7691794-3f77-4418-a425-bb982ecdf587\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:26:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1692c1e7-b315-49c4-b734-7f9f73c715d8, 5adc42a6-f6c9-4de1-a2ff-c199d1ea9530\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:31:57 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1026b587-0a22-4800-88d9-3c200055879f, 406d668f-7c1d-4b64-bff9-7f2bfa725f47\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:36:54 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'23328c29-0dcb-4af3-897e-a48d0601d091, 28b0be28-9d7d-49ea-8e40-73ebdc371b9f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:42:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'853aec7f-a4e2-4930-b433-49328f206465, d0130936-eae7-4c56-9e38-3045bd0a0f77\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:47:01 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d4791b5b-cfd0-4743-91ec-29b68ad0e24c, 446895ad-9855-4ebf-ae94-83f888ab5357\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:52:04 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'56c1c23f-f36a-4146-890e-28a29b63da26, 13a38c8e-a486-4ad8-88e8-965cfc5690f7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:57:05 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0b9afc2f-2490-43c1-8645-a5d415538940, db5e2d06-bdaf-48aa-a35f-a7dceacb2538\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:02:20 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'541f5921-430c-41e4-bfac-2576ad888f63, 0bef74a9-00e3-4c03-889f-cad4f4cec61b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:07:06 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'adea495c-f143-4e31-9ff8-ca9f8aca5799, 2f683007-d22a-4ea3-b8af-90a30524cbfd\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:12:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0138cbb0-23cc-4504-9ce6-5bf478bd96ef, 6d8e507c-7792-47d2-8d39-88fce17fe72c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:17:09 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d143d624-c851-4bfa-8a35-02d6298b008f, 9748bcdd-6997-4cb6-9bd6-2d3ac5df2834\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:22:13 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of zonemercy-vingt-deux-v6-2e6_v4
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
%s, retrying in %s seconds...
Checking if service zonemercy-vingt-deux-v6-2e6-v4 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'367ef51e-f913-4900-81a3-d70fa3171c2b, 0db13607-69e1-46c1-b8ac-1f43c2d9187f\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:27:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
zonemercy-vingt-deux-v6-2e6_v4 status is now torndown due to DeploymentManager action