Deleting key chaiml-0918-horror-retor-3693-v4/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key chaiml-0919-virgo-top-bo-1691-v1/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key chaiml-0919-virgo-top-bo-9296-v1/tokenizer_config.json from bucket guanaco-mkml-models
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Pipeline stage MKMLModelDeleter completed in 8.65s
Pipeline stage MKMLModelDeleter completed in 7.70s
Running pipeline stage MKMLDeleter
Pipeline stage MKMLModelDeleter completed in 7.59s
run pipeline %s
Shutdown handler de-registered
Checking if service chaiml-albert-dpo-0912-v-5607-v2 is running
Shutdown handler de-registered
chaiml-0918-horror-retor_3693_v4 status is now torndown due to DeploymentManager action
chaiml-0919-virgo-top-bo_9296_v1 status is now torndown due to DeploymentManager action
Running pipeline stage MKMLizer
Tearing down inference service chaiml-albert-dpo-0912-v-5607-v2
Starting job with name chaiml-nemo-ties-esao-albert-v3-mkmlizer
Service chaiml-albert-dpo-0912-v-5607-v2 has been torndown
Waiting for job on chaiml-nemo-ties-esao-albert-v3-mkmlizer to finish
Pipeline stage MKMLDeleter completed in 3.38s
run pipeline stage %s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-albert-dpo-0912-v-5607-v2/config.json from bucket guanaco-mkml-models
Deleting key chaiml-albert-dpo-0912-v-5607-v2/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key chaiml-albert-dpo-0912-v-5607-v2/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-albert-dpo-0912-v-5607-v2/tokenizer.json from bucket guanaco-mkml-models
Deleting key chaiml-albert-dpo-0912-v-5607-v2/tokenizer_config.json from bucket guanaco-mkml-models
Pipeline stage MKMLModelDeleter completed in 1.89s
Shutdown handler de-registered
chaiml-albert-dpo-0912-v_5607_v2 status is now torndown due to DeploymentManager action
admin requested tearing down of chaiml-elite-feed-convo-_6421_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
admin requested tearing down of chaiml-elite-feed-convo-_6421_v2
run pipeline %s
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ _____ __ __ ║
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline stage %s
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
admin requested tearing down of chaiml-elite-feed-convo-_7085_v4
run pipeline %s
Running pipeline stage MKMLDeleter
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of mistralai-mistral-small-_5341_v1
Checking if service chaiml-elite-feed-convo-6421-v1 is running
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
Running pipeline stage MKMLDeleter
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ /___/ ║
run pipeline stage %s
Checking if service chaiml-elite-feed-convo-6421-v2 is running
run pipeline %s
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ ║
Running pipeline stage MKMLDeleter
run pipeline stage %s
Tearing down inference service chaiml-elite-feed-convo-6421-v1
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ Version: 0.11.12 ║
Tearing down inference service chaiml-elite-feed-convo-6421-v2
Running pipeline stage MKMLDeleter
Checking if service chaiml-elite-feed-convo-7085-v4 is running
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
Service chaiml-elite-feed-convo-6421-v1 has been torndown
Checking if service mistralai-mistral-small-5341-v1 is running
Service chaiml-elite-feed-convo-6421-v2 has been torndown
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ https://mk1.ai ║
Pipeline stage MKMLDeleter completed in 5.32s
Pipeline stage MKMLDeleter completed in 3.85s
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ ║
run pipeline stage %s
run pipeline stage %s
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ The license key for the current software has been verified as ║
Tearing down inference service chaiml-elite-feed-convo-7085-v4
Running pipeline stage MKMLModelDeleter
Running pipeline stage MKMLModelDeleter
Tearing down inference service mistralai-mistral-small-5341-v1
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ belonging to: ║
Service chaiml-elite-feed-convo-7085-v4 has been torndown
Cleaning model data from S3
Cleaning model data from S3
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ ║
Service mistralai-mistral-small-5341-v1 has been torndown
Pipeline stage MKMLDeleter completed in 5.52s
Cleaning model data from model cache
Cleaning model data from model cache
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ Chai Research Corp. ║
run pipeline stage %s
Pipeline stage MKMLDeleter completed in 5.64s
Deleting key chaiml-elite-feed-convo-6421-v1/config.json from bucket guanaco-mkml-models
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
Running pipeline stage MKMLModelDeleter
Deleting key chaiml-elite-feed-convo-6421-v2/config.json from bucket guanaco-mkml-models
run pipeline stage %s
Deleting key chaiml-elite-feed-convo-6421-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
Cleaning model data from S3
Deleting key chaiml-elite-feed-convo-6421-v2/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Running pipeline stage MKMLModelDeleter
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ║ ║
Cleaning model data from model cache
Cleaning model data from S3
chaiml-nemo-ties-esao-albert-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Deleting key chaiml-elite-feed-convo-7085-v4/config.json from bucket guanaco-mkml-models
Deleting key chaiml-elite-feed-convo-6421-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-elite-feed-convo-6421-v2/special_tokens_map.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key chaiml-elite-feed-convo-7085-v4/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key chaiml-elite-feed-convo-6421-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key chaiml-elite-feed-convo-6421-v2/tokenizer.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-small-5341-v1/config.json from bucket guanaco-mkml-models
Deleting key chaiml-elite-feed-convo-6421-v1/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key chaiml-elite-feed-convo-6421-v2/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-small-5341-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Pipeline stage MKMLModelDeleter completed in 9.44s
Pipeline stage MKMLModelDeleter completed in 9.45s
Shutdown handler de-registered
Deleting key chaiml-elite-feed-convo-7085-v4/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-small-5341-v1/flywheel_model.1.safetensors from bucket guanaco-mkml-models
Shutdown handler de-registered
chaiml-elite-feed-convo-_6421_v1 status is now torndown due to DeploymentManager action
Deleting key chaiml-elite-feed-convo-7085-v4/tokenizer.json from bucket guanaco-mkml-models
chaiml-elite-feed-convo-_6421_v2 status is now torndown due to DeploymentManager action
Deleting key mistralai-mistral-small-5341-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-elite-feed-convo-7085-v4/tokenizer_config.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-small-5341-v1/tokenizer.json from bucket guanaco-mkml-models
Pipeline stage MKMLModelDeleter completed in 8.39s
Deleting key mistralai-mistral-small-5341-v1/tokenizer.model from bucket guanaco-mkml-models
Shutdown handler de-registered
Deleting key mistralai-mistral-small-5341-v1/tokenizer_config.json from bucket guanaco-mkml-models
chaiml-elite-feed-convo-_7085_v4 status is now torndown due to DeploymentManager action
Pipeline stage MKMLModelDeleter completed in 8.29s
Shutdown handler de-registered
mistralai-mistral-small-_5341_v1 status is now torndown due to DeploymentManager action
chaiml-nemo-ties-esao-albert-v3-mkmlizer: Downloaded to shared memory in 48.054s
chaiml-nemo-ties-esao-albert-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpc1ow65hf, device:0
chaiml-nemo-ties-esao-albert-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nemo-ties-esao-albert-v3-mkmlizer: quantized model in 35.501s
chaiml-nemo-ties-esao-albert-v3-mkmlizer: Processed model ChaiML/nemo-ties-esao-albert in 83.555s
chaiml-nemo-ties-esao-albert-v3-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nemo-ties-esao-albert-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nemo-ties-esao-albert-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nemo-ties-esao-albert-v3
chaiml-nemo-ties-esao-albert-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nemo-ties-esao-albert-v3/config.json
chaiml-nemo-ties-esao-albert-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nemo-ties-esao-albert-v3/special_tokens_map.json
chaiml-nemo-ties-esao-albert-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nemo-ties-esao-albert-v3/tokenizer_config.json
chaiml-nemo-ties-esao-albert-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-nemo-ties-esao-albert-v3/tokenizer.json
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-nemo-dare-linear-4360-v2-mkmlizer
Waiting for job on chaiml-nemo-dare-linear-4360-v2-mkmlizer to finish
chaiml-nemo-ties-esao-albert-v3-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nemo-ties-esao-albert-v3/flywheel_model.0.safetensors
chaiml-nemo-ties-esao-albert-v3-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 2/363 [00:06<18:12, 3.03s/it]
Loading 0: 2%|▏ | 6/363 [00:06<04:51, 1.23it/s]
Loading 0: 4%|▎ | 13/363 [00:06<01:44, 3.37it/s]
Loading 0: 5%|▍ | 17/363 [00:06<01:10, 4.92it/s]
Loading 0: 6%|▋ | 23/363 [00:06<00:42, 8.02it/s]
Loading 0: 8%|▊ | 29/363 [00:06<00:29, 11.45it/s]
Loading 0: 9%|▉ | 34/363 [00:06<00:22, 14.79it/s]
Loading 0: 11%|█ | 40/363 [00:07<00:19, 16.49it/s]
Loading 0: 12%|█▏ | 44/363 [00:07<00:16, 19.13it/s]
Loading 0: 14%|█▍ | 50/363 [00:07<00:12, 24.64it/s]
Loading 0: 15%|█▌ | 56/363 [00:07<00:11, 27.87it/s]
Loading 0: 17%|█▋ | 61/363 [00:07<00:10, 29.93it/s]
Loading 0: 18%|█▊ | 67/363 [00:07<00:08, 35.56it/s]
Loading 0: 20%|█▉ | 72/363 [00:07<00:07, 37.80it/s]
Loading 0: 21%|██ | 77/363 [00:07<00:07, 39.75it/s]
Loading 0: 23%|██▎ | 83/363 [00:08<00:07, 39.18it/s]
Loading 0: 24%|██▍ | 88/363 [00:08<00:06, 39.63it/s]
Loading 0: 26%|██▌ | 94/363 [00:08<00:06, 43.86it/s]
Loading 0: 27%|██▋ | 99/363 [00:08<00:05, 44.01it/s]
Loading 0: 29%|██▊ | 104/363 [00:08<00:05, 44.85it/s]
Loading 0: 30%|███ | 110/363 [00:08<00:05, 42.28it/s]
Loading 0: 32%|███▏ | 115/363 [00:08<00:05, 41.76it/s]
Loading 0: 33%|███▎ | 121/363 [00:09<00:07, 31.89it/s]
Loading 0: 34%|███▍ | 125/363 [00:09<00:07, 32.47it/s]
Loading 0: 36%|███▌ | 131/363 [00:09<00:06, 37.36it/s]
Loading 0: 38%|███▊ | 137/363 [00:09<00:05, 38.19it/s]
Loading 0: 39%|███▉ | 142/363 [00:09<00:05, 38.96it/s]
Loading 0: 41%|████ | 149/363 [00:09<00:04, 44.38it/s]
Loading 0: 43%|████▎ | 155/363 [00:09<00:04, 42.74it/s]
Loading 0: 44%|████▍ | 160/363 [00:10<00:04, 42.32it/s]
Loading 0: 46%|████▌ | 167/363 [00:10<00:04, 47.31it/s]
Loading 0: 48%|████▊ | 173/363 [00:10<00:04, 44.73it/s]
Loading 0: 49%|████▉ | 178/363 [00:10<00:04, 43.55it/s]
Loading 0: 51%|█████ | 185/363 [00:10<00:03, 47.87it/s]
Loading 0: 53%|█████▎ | 191/363 [00:10<00:03, 45.14it/s]
Loading 0: 54%|█████▍ | 196/363 [00:10<00:03, 44.35it/s]
Loading 0: 56%|█████▌ | 202/363 [00:11<00:04, 34.63it/s]
Loading 0: 57%|█████▋ | 206/363 [00:11<00:04, 35.19it/s]
Loading 0: 58%|█████▊ | 212/363 [00:11<00:03, 39.36it/s]
Loading 0: 60%|██████ | 218/363 [00:11<00:03, 38.93it/s]
Loading 0: 61%|██████▏ | 223/363 [00:11<00:03, 39.38it/s]
Loading 0: 63%|██████▎ | 230/363 [00:11<00:02, 44.71it/s]
Loading 0: 65%|██████▌ | 236/363 [00:11<00:02, 42.95it/s]
Loading 0: 66%|██████▋ | 241/363 [00:11<00:02, 42.35it/s]
Loading 0: 68%|██████▊ | 247/363 [00:12<00:02, 46.32it/s]
Loading 0: 69%|██████▉ | 252/363 [00:12<00:02, 46.06it/s]
Loading 0: 71%|███████ | 257/363 [00:12<00:02, 47.02it/s]
Loading 0: 72%|███████▏ | 263/363 [00:12<00:02, 44.34it/s]
Loading 0: 74%|███████▍ | 268/363 [00:12<00:02, 42.96it/s]
Loading 0: 75%|███████▌ | 274/363 [00:12<00:01, 47.04it/s]
Loading 0: 77%|███████▋ | 279/363 [00:12<00:01, 46.91it/s]
Loading 0: 78%|███████▊ | 284/363 [00:13<00:02, 33.00it/s]
Loading 0: 80%|███████▉ | 290/363 [00:13<00:02, 34.79it/s]
Loading 0: 81%|████████ | 294/363 [00:13<00:01, 34.69it/s]
Loading 0: 83%|████████▎ | 301/363 [00:13<00:01, 42.07it/s]
Loading 0: 84%|████████▍ | 306/363 [00:13<00:01, 41.86it/s]
Loading 0: 86%|████████▌ | 311/363 [00:13<00:01, 43.03it/s]
Loading 0: 87%|████████▋ | 316/363 [00:13<00:01, 44.38it/s]
Loading 0: 88%|████████▊ | 321/363 [00:13<00:01, 37.15it/s]
Loading 0: 90%|█████████ | 328/363 [00:14<00:00, 44.65it/s]
Loading 0: 92%|█████████▏| 333/363 [00:14<00:00, 43.42it/s]
Loading 0: 93%|█████████▎| 338/363 [00:14<00:00, 43.75it/s]
Loading 0: 95%|█████████▍| 344/363 [00:14<00:00, 41.94it/s]
Loading 0: 96%|█████████▌| 349/363 [00:14<00:00, 41.51it/s]
Loading 0: 98%|█████████▊| 356/363 [00:14<00:00, 46.21it/s]
Loading 0: 100%|█████████▉| 362/363 [00:14<00:00, 43.58it/s]
Job chaiml-nemo-ties-esao-albert-v3-mkmlizer completed after 109.12s with status: succeeded
Stopping job with name chaiml-nemo-ties-esao-albert-v3-mkmlizer
Pipeline stage MKMLizer completed in 109.78s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 3.78s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nemo-ties-esao-albert-v3
Waiting for inference service chaiml-nemo-ties-esao-albert-v3 to be ready
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ _____ __ __ ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ /___/ ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ Version: 0.11.12 ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ https://mk1.ai ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ belonging to: ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ Chai Research Corp. ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ║ ║
chaiml-nemo-dare-linear-4360-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
admin requested tearing down of blend_rofur_2024-10-03
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of mistralai-mistral-small_5341_v38
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
admin requested tearing down of chaiml-0916-intent-suppo_6584_v3
run pipeline stage %s
chaiml-nemo-dare-linear-4360-v2-mkmlizer: Downloaded to shared memory in 48.061s
run pipeline %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage ProductionBlendMKMLTemplater
chaiml-nemo-dare-linear-4360-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpiuzzjusy, device:0
run pipeline stage %s
run pipeline %s
Pipeline stage %s skipped, reason=%s
chaiml-nemo-dare-linear-4360-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Running pipeline stage MKMLDeleter
run pipeline stage %s
Pipeline stage ProductionBlendMKMLTemplater completed in 12.74s
Skipping teardown as no inference service was successfully deployed
Running pipeline stage MKMLDeleter
run pipeline stage %s
Pipeline stage MKMLDeleter completed in 17.29s
chaiml-nemo-dare-linear-4360-v2-mkmlizer: quantized model in 37.119s
Pipeline stage %s skipped, reason=%s
Running pipeline stage MKMLDeployer
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline stage %s
chaiml-nemo-dare-linear-4360-v2-mkmlizer: Processed model ChaiML/nemo-dare-linear-esao-albert in 85.180s
Pipeline stage MKMLDeleter completed in 13.32s
run pipeline %s
Creating inference service blend-rofur-2024-10-03
Running pipeline stage MKMLModelDeleter
chaiml-nemo-dare-linear-4360-v2-mkmlizer: creating bucket guanaco-mkml-models
run pipeline stage %s
run pipeline stage %s
Skipping deletion as no model was successfully uploaded
Ignoring service blend-rofur-2024-10-03 already deployed
chaiml-nemo-dare-linear-4360-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
Running pipeline stage MKMLModelDeleter
Running pipeline stage MKMLizer
Pipeline stage MKMLModelDeleter completed in 17.54s
Waiting for inference service blend-rofur-2024-10-03 to be ready
chaiml-nemo-dare-linear-4360-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nemo-dare-linear-4360-v2
Cleaning model data from S3
Starting job with name chaiml-nemo-ties-lexical-2876-v2-mkmlizer
Inference service chaiml-nemo-ties-esao-albert-v3 ready after 120.59382796287537s
Shutdown handler de-registered
chaiml-nemo-dare-linear-4360-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nemo-dare-linear-4360-v2/config.json
Cleaning model data from model cache
Waiting for job on chaiml-nemo-ties-lexical-2876-v2-mkmlizer to finish
Pipeline stage MKMLDeployer completed in 131.07s
mistralai-mistral-small_5341_v38 status is now torndown due to DeploymentManager action
chaiml-nemo-dare-linear-4360-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nemo-dare-linear-4360-v2/special_tokens_map.json
Deleting key chaiml-0916-intent-suppo-6584-v3/tokenizer_config.json from bucket guanaco-mkml-models
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
run pipeline stage %s
chaiml-nemo-dare-linear-4360-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nemo-dare-linear-4360-v2/tokenizer_config.json
Pipeline stage MKMLModelDeleter completed in 28.51s
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ _____ __ __ ║
Running pipeline stage StressChecker
Shutdown handler de-registered
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
chaiml-0916-intent-suppo_6584_v3 status is now torndown due to DeploymentManager action
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
Job chaiml-nemo-dare-linear-4360-v2-mkmlizer completed after 164.31s with status: succeeded
Received healthy response to inference request in 3.2460777759552s
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
Stopping job with name chaiml-nemo-dare-linear-4360-v2-mkmlizer
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ /___/ ║
Pipeline stage MKMLizer completed in 174.00s
Received healthy response to inference request in 3.152304172515869s
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ ║
run pipeline stage %s
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ Version: 0.11.12 ║
Received healthy response to inference request in 1.8330910205841064s
Running pipeline stage MKMLTemplater
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
Pipeline stage MKMLTemplater completed in 5.13s
Received healthy response to inference request in 2.05292010307312s
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ https://mk1.ai ║
run pipeline stage %s
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ ║
Received healthy response to inference request in 1.856792688369751s
Running pipeline stage MKMLDeployer
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ The license key for the current software has been verified as ║
5 requests
Creating inference service chaiml-nemo-dare-linear-4360-v2
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ belonging to: ║
0 failed requests
Waiting for inference service chaiml-nemo-dare-linear-4360-v2 to be ready
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ ║
5th percentile: 1.8378313541412354
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ Chai Research Corp. ║
10th percentile: 1.8425716876983642
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
20th percentile: 1.852052354812622
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
30th percentile: 1.8960181713104247
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ║ ║
40th percentile: 1.9744691371917724
chaiml-nemo-ties-lexical-2876-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
50th percentile: 2.05292010307312
60th percentile: 2.49267373085022
70th percentile: 2.932427358627319
80th percentile: 3.1710588932037354
90th percentile: 3.2085683345794678
95th percentile: 3.227323055267334
Job chaiml-nemo-ties-lexical-2876-v2-mkmlizer completed after 119.97s with status: succeeded
99th percentile: 3.242326831817627
Stopping job with name chaiml-nemo-ties-lexical-2876-v2-mkmlizer
mean time: 2.4282371520996096
Pipeline stage MKMLizer completed in 147.06s
Pipeline stage StressChecker completed in 118.12s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Running pipeline stage TriggerMKMLProfilingPipeline
Pipeline stage MKMLTemplater completed in 3.32s
run_pipeline:run_in_cloud %s
run pipeline stage %s
starting trigger_guanaco_pipeline args=%s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nemo-ties-lexical-2876-v2
Waiting for inference service chaiml-nemo-ties-lexical-2876-v2 to be ready
Pipeline stage TriggerMKMLProfilingPipeline completed in 13.39s
Shutdown handler de-registered
chaiml-nemo-ties-esao-albert_v3 status is now deployed due to DeploymentManager action
chaiml-nemo-ties-esao-albert_v3 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6684dc92-2689-4fda-8043-d68d7e875d57, 42fcc605-2524-4440-94c3-753794a423c7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:15:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9d5c9596-57c5-4e46-a37a-e36d5c65f6dc, 2dc790f6-6d09-4345-a4b5-d4b5248fcdf0\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:20:39 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7adc8f25-42c9-46b2-afae-ed25b2017f4d, 2c9b92f5-af95-4754-aa5d-47e557e752b6\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:25:41 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'59439388-ed65-437d-a9a5-7aa0d4b8f4b0, 4396f12a-38d9-4b59-adc2-164339ddd45d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:30:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'499684e3-6dac-4a8b-b733-20ec46674e1c, f00e95b5-3f57-4bb8-be75-bc4af9f8a447\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:35:43 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'86546c16-2c5f-4e86-abb3-0eb8617f84b0, 09b652ed-7bc4-4b8b-b0a1-f88785fd668d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:40:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d97761ff-8839-4fce-a5b7-6243032efce0, a2a7f27f-bbe5-4e98-9cdd-6c836c3e02b7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:45:46 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1ffb3890-8c5e-4bd9-8f18-29054d613bb0, 3ec8b453-79be-4e43-8c10-e5ff5c9a04f7\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:50:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'75491e95-8178-49d6-9681-bb5924508e75, 7f8ce52b-5bd4-47df-92cf-51b54fcff45c\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 14:55:45 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b4531548-c905-46f4-b52d-ce4dcd2fa9ea, e829a63a-2494-4257-9584-dca3fb2b7efa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:00:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ac66e5b5-503f-455a-89f3-4ffa0c51b4f0, ee8dfa5d-e6e2-4beb-a9fa-843784550ae5\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:05:49 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2a640697-f548-4417-bddf-631364e70744, d423ace0-b9ce-4a32-9a45-541c53293e15\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:10:48 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f8d36a9a-1f93-4a71-9960-327cd1002be1, e01ad40d-2e84-4194-b063-9933d5fdf895\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:15:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
admin requested tearing down of chaiml-nemo-ties-lexical_2876_v3
%s, retrying in %s seconds...
Shutdown handler not registered because Python interpreter is not running in the main thread
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
run pipeline %s
%s, retrying in %s seconds...
run pipeline stage %s
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
Running pipeline stage MKMLDeleter
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'32bc31e2-e953-44b2-84a6-25f727a8966f, 445a6609-0baa-4257-968b-92be091b39aa\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:20:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Checking if service chaiml-nemo-ties-lexical-2876-v3 is running
Shutdown handler de-registered
%s, retrying in %s seconds...
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'295a5ef7-21dd-4b36-8812-75ba19676ead, 7a6207f7-f2e9-46d6-bc43-16c98a67f228\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:25:50 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'449d988e-59eb-4974-8c16-02f4cc924a2b, 8318e195-86ae-47ae-9cfc-8f4510a4bd12\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:30:52 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0c7734e4-60cc-49b9-a1cd-b9f738a44739, c4986f9e-827d-4a27-80cc-4f3fcb8a7214\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:35:51 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8072535e-84dd-4374-afad-fb3e7c8fdb49, 9ac24e20-afb3-4517-b00d-69f0ffa2b4d1\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:40:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7b2c8398-b8d3-4f9c-add4-e8c4d7bc57f1, 31c0714d-02f7-4391-93a1-4a572d7a9d91\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:45:55 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d5353f43-39e6-4470-8c34-ed70bd57f918, e200c2da-8cea-4d47-b139-b82625c538a2\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:50:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1440b750-9542-4fec-a71a-e192e304194d, 3e3293dc-9074-41c7-a261-165edc6a3e63\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 15:55:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a06e245b-f873-4c54-8776-814b5aab26cf, cc0b2fce-a864-4d85-ac9a-af569b8d2691\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:01:10 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5656e2e5-9af7-421c-b5c9-5cceb95e15f9, e0ae8910-9b29-41ba-a466-2bf61afd9405\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:05:56 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5aa68b5a-e5a2-412d-a1ff-7e7db9535788, eddeb202-70ff-49a2-bd49-fbe4bb31743b\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:10:59 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'620caf22-d005-46a0-b0a2-1515e6d19549, aa8aca97-7270-402d-a847-9a18dd721261\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:15:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bee57714-8567-4e52-971b-6c94ef52e665, 5f940abe-184c-4f38-8bf7-b3d7178bb340\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:21:00 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nemo-ties-esao-albert_v3
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
%s, retrying in %s seconds...
Checking if service chaiml-nemo-ties-esao-albert-v3 is running
clean up pipeline due to error=TeardownError('Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'69aff0fd-22d0-4f21-a023-f7422af223ea, 208630cf-2d5a-4cd1-908b-12ce6065f0bf\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'406\', \'Content-Type\': \'application/json\', \'Date\': \'Mon, 14 Oct 2024 16:25:58 GMT\', \'X-Content-Type-Options\': \'nosniff\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'33f90070-eb4d-4b82-b71a-96fabdf69ad9\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'07a050f1-ad60-4b87-8572-f4584856b92a\'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io is forbidden: User \\"system:serviceaccount:guanaco-backend:default\\" cannot list resource \\"inferenceservices\\" in API group \\"serving.kserve.io\\" in the namespace \\"tenant-chaiml-guanaco\\"","reason":"Forbidden","details":{"group":"serving.kserve.io","kind":"inferenceservices"},"code":403}\n\n\n')
Shutdown handler de-registered
chaiml-nemo-ties-esao-albert_v3 status is now torndown due to DeploymentManager action