chaiml-cogito32b-dpos2-_39363

developer_uid: richhx
submission_id: chaiml-cogito32b-dpos2-_39363_v9
model_name: chaiml-cogito32b-dpos2-_39363_v9
model_group: ChaiML/cogito32b-dpos2-d
status: inactive
timestamp: 2025-08-05T16:35:40+00:00
num_battles: 5621
num_wins: 2821
celo_rating: 1262.91
family_friendly_score: 0.4638
family_friendly_standard_error: 0.007052511042174979
submission_type: basic
model_repo: ChaiML/cogito32b-dpos2-dpos1ft-b06-2e
model_architecture: Qwen2ForCausalLM
model_num_parameters: 32759331840.0
best_of: 5
max_input_tokens: 768
max_output_tokens: 64
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.41714531882372485, 'latency_mean': 2.3971461164951324, 'latency_p50': 2.391227126121521, 'latency_p90': 2.6672103881835936}, {'batch_size': 3, 'throughput': 0.9098032921607336, 'latency_mean': 3.2796036267280577, 'latency_p50': 3.292022943496704, 'latency_p90': 3.639439582824707}, {'batch_size': 5, 'throughput': 1.2086360097173343, 'latency_mean': 4.114807615280151, 'latency_p50': 4.124342441558838, 'latency_p90': 4.560865330696106}, {'batch_size': 6, 'throughput': 1.304020458380429, 'latency_mean': 4.563847805261612, 'latency_p50': 4.592350721359253, 'latency_p90': 5.075848650932312}, {'batch_size': 8, 'throughput': 1.4640598681409454, 'latency_mean': 5.396030632257461, 'latency_p50': 5.381523013114929, 'latency_p90': 6.074738907814026}, {'batch_size': 10, 'throughput': 1.5841524415867383, 'latency_mean': 6.23944675207138, 'latency_p50': 6.270308136940002, 'latency_p90': 7.010861039161682}]
gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}
display_name: chaiml-cogito32b-dpos2-_39363_v9
ineligible_reason: num_battles<10000
is_internal_developer: True
language_model: ChaiML/cogito32b-dpos2-dpos1ft-b06-2e
model_size: 33B
ranking_group: single
throughput_3p7s: 1.08
us_pacific_date: 2025-08-05
win_ratio: 0.50186799501868
generation_params: {'temperature': 0.5, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 40, 'presence_penalty': 0.35, 'frequency_penalty': 0.35, 'stopping_words': ['\n'], 'max_input_tokens': 768, 'best_of': 5, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-cogito32b-dpos2-39363-v9-mkmlizer
Waiting for job on chaiml-cogito32b-dpos2-39363-v9-mkmlizer to finish
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║                                                                     ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║                                                                     ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  Version: 0.29.15                                                   ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║                                                                     ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  belonging to:                                                      ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║                                                                     ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ║                                                                     ║
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: Downloaded to shared memory in 82.497s
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: Checking if ChaiML/cogito32b-dpos2-dpos1ft-b06-2e already exists in ChaiML
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmp_ikfuwmw, device:0
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: Saving flywheel model at /dev/shm/model_cache
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-biker-twice-dark_19320_v1: HTTPConnectionPool(host='chaiml-biker-twice-dark-19320-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: quantized model in 448.378s
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: Processed model ChaiML/cogito32b-dpos2-dpos1ft-b06-2e in 530.876s
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: creating bucket guanaco-mkml-models
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v9/nvidia
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v9/nvidia/config.json
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v9/nvidia/added_tokens.json
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v9/nvidia/tokenizer_config.json
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v9/nvidia/special_tokens_map.json
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: cp /dev/shm/model_cache/merges.txt s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v9/nvidia/merges.txt
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: cp /dev/shm/model_cache/vocab.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v9/nvidia/vocab.json
chaiml-cogito32b-dpos2-39363-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v9/nvidia/tokenizer.json
Job chaiml-cogito32b-dpos2-39363-v9-mkmlizer completed after 563.57s with status: succeeded
Stopping job with name chaiml-cogito32b-dpos2-39363-v9-mkmlizer
Pipeline stage MKMLizer completed in 564.51s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-cogito32b-dpos2-39363-v9
Waiting for inference service chaiml-cogito32b-dpos2-39363-v9 to be ready
Failed to get response for submission chaiml-biker-twice-dark_19320_v1: HTTPConnectionPool(host='chaiml-biker-twice-dark-19320-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-cogito32b-dpos2-39363-v9 ready after 312.6397604942322s
Pipeline stage MKMLDeployer completed in 313.67s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.093339443206787s
Received healthy response to inference request in 2.39096736907959s
Received healthy response to inference request in 2.5858993530273438s
Received healthy response to inference request in 2.5648093223571777s
Received healthy response to inference request in 3.757223606109619s
5 requests
0 failed requests
5th percentile: 2.4257357597351072
10th percentile: 2.460504150390625
20th percentile: 2.5300409317016603
30th percentile: 2.569027328491211
40th percentile: 2.5774633407592775
50th percentile: 2.5858993530273438
60th percentile: 2.788875389099121
70th percentile: 2.9918514251708985
80th percentile: 3.2261162757873536
90th percentile: 3.491669940948486
95th percentile: 3.6244467735290526
99th percentile: 3.730668239593506
mean time: 2.8784478187561033
Pipeline stage StressChecker completed in 15.82s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.74s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.65s
Shutdown handler de-registered
chaiml-cogito32b-dpos2-_39363_v9 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3007.69s
Shutdown handler de-registered
chaiml-cogito32b-dpos2-_39363_v9 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c35b5d91-e631-49ea-8c5c-32ae7a70b131\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:15:34 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f49fafce-a982-443a-a9c4-54a025d6630f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:20:17 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'87d59bd8-ff31-4ffb-897c-b8cf974d0f44\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:25:19 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9f39122d-a93c-41ac-a141-0b28cf6de19a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:30:14 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'68fdaeed-5317-43da-9e6e-60cdd7324b6c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:35:14 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8058625c-14d2-47ff-baad-de07741c7bea\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:40:14 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7f632775-7390-47bc-9448-ad9b89bae53b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:45:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0f8502f4-6884-40ab-a0b9-092707d41a03\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:50:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b07e519a-9873-4507-8acb-a6e6f2325f7a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:55:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2f18b0b0-3e32-4fde-993b-bb0e385b7ac4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:00:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ae91aec1-d9eb-4add-9091-68bfbfc79eb1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:05:18 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9bffe3cf-a1ee-431f-a022-c63f9bea6243\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:10:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5bdcd337-494f-4d8b-81a3-55409fe68a47\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:15:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f8b4d8d7-5566-4cde-a945-8fbf84a004d0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:20:29 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'edb382b4-62c9-4f14-a22d-4eb6fb8b3ea0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:25:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7c4cb9bc-cc62-48ed-a3e7-b7e9307eae07\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:30:29 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'af2786bd-9525-45f9-8995-2e348c39f6ae\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:35:29 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'034e757a-3a08-4037-961b-0bb40aae65bb\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:40:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c840c246-739a-4c0b-8f34-a205744d60ed\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:45:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'87017ac0-0abb-438b-9228-a931699d28c3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:50:21 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'17664763-7303-4b28-91cb-22a8f0ddbc5e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:55:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'736cda97-1377-4140-b7b1-e8fa93103eb9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:00:24 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'493ced4c-d849-43bf-b699-1c644fd32247\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:05:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'54bb6562-6d2a-49f7-a4b8-cbcb65ed0604\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:10:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b9efaa38-6a74-4ad1-8c92-52ae319757f4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:15:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a2464ad5-fafc-42ae-8741-75944df2523e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:20:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2c74ed0c-1084-46b0-a5dd-020336c6043a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:25:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'00990ce3-ce42-4369-a562-e0e1676c305d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:30:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1c877185-cfe2-4005-a2b4-5a6ba4139c5f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:35:28 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'51634e62-4860-40f9-8152-ff715dcd9ea1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:40:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c3dffa66-6f91-4f5e-9b12-f05ec167281d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:45:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'62cccd6b-70b0-49e1-9dd7-bf8e0233b1de\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:50:21 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'88dc2783-bdb1-433a-8f53-8cfc662d914d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:55:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a4912a1a-f17c-4177-b865-3227f4949df6\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:00:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d76dcadc-f25c-4a93-93da-f01b6516772c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:05:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2c4912f-7b55-4d15-b53e-25fdc9321af5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:10:26 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fd221b71-1716-41e6-b765-dff3b6eef5a9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:15:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'461a7b74-11ce-40f9-a459-af225cde8fde\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:20:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'37f9f4e0-5610-4c81-ad29-6675b6cc7634\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:25:24 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5c4ce680-59f5-42bd-bdca-71d5b6c6edd5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:30:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a092d96c-fd67-4c66-bfc5-2e771e17c879\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:35:24 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'19470a63-c752-42cc-95cf-a8367bd26844\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:40:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'afa24136-a373-44db-9bcc-f38acb7d89f0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:45:24 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'680f7535-8362-4478-beb8-3955101b8b0a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:50:29 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c9346cb6-67fc-446f-9a70-6503f8649905\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:55:24 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ce251875-aba2-460b-9fbb-9dc80892c157\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:00:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fd70ae29-ff0b-4e86-ad16-0225cd8c88d3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:05:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'020fe737-9c2d-4ecb-9d4a-80386fe35ded\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:10:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'62a33a2d-2c93-4dff-9d5e-20a89fa2f315\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:15:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6c06dde3-5e99-4a36-add6-1a16c69284c0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:20:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cec1e9a6-1072-42e2-91d7-2149c7afef73\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:25:29 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'23b606d3-6533-4b3f-9c00-28a52a4086fe\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:30:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'43606ba5-b424-490b-a10d-eb77ea871f73\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:35:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0171712a-df92-4e81-b673-0ed464cb31ce\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:40:28 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5015a12e-a058-469a-b07e-5b1d62a949d5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:45:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'239fdf96-9f1c-4c5c-af96-a22aed44a832\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:50:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'670d1c9f-da04-4cd2-987c-26e47bab30bf\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:55:47 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cda03d74-5124-4676-bd6f-4a0bc4ed0809\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:00:42 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a243b7a2-6fdb-425e-9eeb-929b1f021daf\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:05:29 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f8c56145-360f-43f0-a11f-fad3ee63242b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:10:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'04eba3bd-f975-48fa-ac15-196a17ea1322\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:15:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'53b2d329-1c29-4b1e-ad36-d125f7fe139e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:20:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'82a0f22b-bdb9-4e32-ad9a-427cee4a564e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:25:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
%s, retrying in %s seconds...
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'31992c9e-e9bc-4cfb-a1b0-32399cdaa834\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:30:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
run pipeline %s
Shutdown handler de-registered
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8f045238-a119-442b-b2f5-2fb93fa4e6c3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:30:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ba9b5828-b081-40e5-b60b-cd62ca5513bb\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:35:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4ace96bf-2b88-4ddf-8bde-1b4480b3c3ba\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:40:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'528231ed-af94-4512-b136-106e6c7ab866\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:45:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b5220dbc-688a-4a7a-b3d9-6a7057768e06\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:50:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
HTTP Request: %s %s "%s %d %s"
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8d2b3a06-71a5-436d-9988-4c954454885c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:55:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'59cc169f-738f-4d15-ab69-85d0e03077d0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:00:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'52f9f567-ad8b-4260-824f-fd69ce55a99b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:05:35 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b9c6f319-d42e-468a-9882-f56353a3c7e8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:10:35 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e8cc4b83-82ee-4850-96f1-be941e8d1285\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:15:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e009e8a8-d431-47f8-88c8-a2c2c694d5a7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:20:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d0ddda19-3cc2-4087-8a94-3939692b24c4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:25:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f4b28c5d-5b4a-4fc6-8ca6-2a16cc9096a9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:30:38 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a83b52f7-9427-462b-9f0c-d2c2fb65e9b2\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:35:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f9fdf59e-15bd-4710-917c-37e1cf96fa22\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:40:36 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e95e544f-3710-4ead-aa2d-18d2b5d6c457\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:45:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'24c268ff-5ba5-44fe-a7d4-a4231ee4ce95\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:50:35 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
HTTP Request: %s %s "%s %d %s"
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4d09671e-34f7-4bfa-9f00-30a67c81a8d9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:55:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5e78731a-8f4c-4db4-90c7-c6cf33ad86ad\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:00:34 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6ceaa089-75ed-4ec7-91a8-84852039d3e6\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:05:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5cde4375-a121-47aa-b66a-b5294116fc4e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:10:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ebd77aa2-9cfd-470a-9b53-f93b1ac8951b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:15:38 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b633eee8-1f52-47c4-a6c2-e98beb11413e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:20:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'be863255-0f87-4a45-b3f0-3d4fec906662\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:25:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'61f9155d-e85f-43c2-840c-bd3ca1eefd16\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:30:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
HTTP Request: %s %s "%s %d %s"
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'98a505eb-bb68-4638-882c-d270522489e1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:36:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'10045e2b-20e4-4f57-b039-16251144e04d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:40:54 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b7418bd5-22fd-4678-88ca-3b1debdd8c35\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:45:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f7d65a1d-0e1a-43c8-b151-78fe0af57185\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:50:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5780bd02-e56a-4f70-a5f2-30b512b58cec\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:55:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3a3135e3-6b43-4d95-88b6-28728f2b800a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:00:35 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'798397be-58f4-43c5-8883-dd587a935f8f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:05:44 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a7c1527c-3771-4912-a05f-3bcffc0597eb\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:10:43 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f7fe8844-1fae-4570-8901-a2f6d2b374be\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:15:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'617da64e-50d7-4076-abd2-18410184c5b7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:20:47 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0f00feb5-b406-4a0e-9c6f-f476ee6d60f1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:25:46 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'529ab8b1-c7ec-4b9f-82d2-a2e0280b0ae1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:30:43 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5bb93858-8c83-4cb7-a53d-d05d65a70cd3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:35:44 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3f6e1514-a73c-4b48-945d-8f4a9c97611f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:40:46 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'978b5b9d-13f4-4b34-a471-7d4e6292709f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:50:46 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'03bc7e1d-7dee-4040-89a9-565c07653f5e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:55:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'924b5af4-c506-4733-a53d-094424f89468\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:00:45 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1714091d-78f0-46de-b3c2-d71070645aac\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:05:47 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2b0eb26e-ebdb-4318-be39-52191a528dc7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:15:45 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9263b72e-b9c9-4ff9-93c4-5e27a03e1c5d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:20:35 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bf2bfee6-a459-4036-9776-9f6ea2990145\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:25:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6c5a32f3-ab25-4537-a1a9-198ebc11d81c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:35:47 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'36ff0c10-f029-4ce5-b73d-9091294eaa9d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:40:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f64c1c28-114f-4ba4-b022-b762743504b0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:45:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'289eedfc-b465-4979-becf-f0aa59caf059\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:55:42 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'55acb2f1-7024-41a4-b17c-be2ab034dfbd\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:00:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a183704b-6b76-40f5-a5a7-0a0d0cf6201c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:06:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0825a02d-0a96-4a6a-b860-371488ef39c5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:10:55 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'06a2d2db-b6e5-4569-a598-d1c1729bfe8c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:15:45 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4232f918-c836-4af4-ade8-f236cadc3e99\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:20:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4b94bd9a-f747-421e-a723-6a4e9ef4edf6\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:30:53 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5464b13f-ea96-4730-82eb-3844c5899614\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:35:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bb967c13-815b-491e-9013-69df59a5f5d3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:45:42 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0e6d9e39-fd1b-4e6c-bc06-830ea337c26b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:50:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dfe5b29d-8492-47ce-a81e-088a7879b19e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:00:50 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2e05e38-5af7-442c-9602-32073f088fb0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:05:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e0bf280c-6110-47ce-8bc0-98e769353579\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:10:55 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0a0912bc-bd4a-4959-8caf-845dc1620028\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:15:49 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'97fd1c2d-fcad-488d-9f23-39a20a84d285\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:20:35 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7ea14bd3-e74c-4ef5-9c00-a7a1d0d402ce\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:25:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'39211b9c-383e-4f87-9f4a-b0dc2b9a9a76\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:30:44 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2956a46-53df-4d40-b29c-937077d52c89\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:35:34 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fcd687a1-6029-4f99-866e-8142a26df78e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:41:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f1e294a4-7bfe-42db-8776-ed2807947ad4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:45:45 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'eb7d39e0-ab74-4b64-873c-5a9a2fac271e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:50:35 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'785b4afd-6eef-420a-b804-821fdd17617a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:56:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e50e5bba-b067-44b2-81cf-f8bba08d0362\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:00:45 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1aa64476-fdec-4ddd-a967-5e441a68f7dd\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:05:36 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'61e9bb9e-5737-46dc-bdd5-68ff9dc109da\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:11:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5e8b5e42-adbb-4ca9-898a-cc531bd42625\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:15:46 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'95a09fb2-9a3c-4a53-8f9f-4ff2580e956b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:20:38 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c47e1945-ebe9-411c-985c-cd8ccbae2974\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:26:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'180fdd5e-9305-4d4b-aa6e-b5f06e871257\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:30:50 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'141d7d37-4098-469f-84ab-6e2534f71cb3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:35:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'512ec881-35bb-4747-b114-39f875d52422\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:41:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fdce7259-462e-40f2-a093-08c92408bb7d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:45:50 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b615f837-5899-45bf-bde8-441da7fa6244\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:50:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'56b72920-47e7-4c55-b7b9-de5eef52fc65\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:56:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'09665d7d-39e6-4b56-b157-bbffc84fd0a5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:00:50 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'08cf09d1-366b-4fb0-bcce-3b30fb2cebd4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:05:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'454e49e3-23b1-4bac-8d90-70d58edaee94\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:11:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4e40d035-2fdc-49ac-8369-5ef4b3ce8149\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:15:48 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'faaa8daa-bad9-49ec-9730-e792d3d916f0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:20:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5156b1b7-a1c8-45ef-8620-489dedb2b2dd\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:26:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2b19beb4-e1c4-4f75-993c-3a7bdacf68ac\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:30:52 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
admin requested tearing down of nothingiisreal-mn-12b-_64584_v10
%s, retrying in %s seconds...
Shutdown handler not registered because Python interpreter is not running in the main thread
%s, retrying in %s seconds...
run pipeline %s
run pipeline stage %s
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'03e267f4-2a29-4ef3-9520-a05464d8812f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:35:42 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Running pipeline stage MKMLDeleter
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dfefed39-1b9d-4e01-9a57-d01201538a76\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:45:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d449d912-3e0e-4cf8-a18f-6e443c771d70\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:50:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'10969fad-1595-4599-b284-c9455d13b0c1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:55:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4a830508-d8f2-4815-bd47-c7f9921b51fe\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:06:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'655efca6-dce4-4a99-915f-95fe41001787\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:10:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d59c95a4-1797-4d42-81a9-61d4e99c0178\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:15:52 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7e7f986f-d263-4efa-a897-468913aa10a0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:25:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'470dbc3c-ed5f-4ca5-99fd-679c49971ce0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:30:48 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a7f2a935-4515-4dfe-bd33-30e52b7f8b57\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:35:49 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7ce435e6-af45-4e9d-b2c4-805f3004e048\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:46:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b6fc0a94-3603-4057-bc05-49d9b74a6e7a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:50:44 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e367ef15-ee54-4022-9c93-cd987a950379\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 15:20:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0afeadd7-0ecd-4ddc-8ebe-e4f5f5045d0c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 15:25:43 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'568ed186-a3e0-40b2-8d44-2a33a0ef4dc4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 16:01:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-cogito32b-dpos2-_39363_v9
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b9eac83f-ba27-4889-8d1d-1f8d79bf177d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 16:16:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n    return self.rest_client.GET(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n    return self.request("GET", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
Shutdown handler de-registered