Shutdown handler not registered because Python interpreter is not running in the main thread
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
run pipeline %s
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
run pipeline stage %s
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Running pipeline stage MKMLizer
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Starting job with name chaiml-mistral32-deeps-89261-v11-mkmlizer
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Waiting for job on chaiml-mistral32-deeps-89261-v11-mkmlizer to finish
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
HTTP Request: %s %s "%s %d %s"
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Version: 0.29.15 ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ https://mk1.ai ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ The license key for the current software has been verified as ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ belonging to: ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Chai Research Corp. ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Version: 0.29.15 ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ https://mk1.ai ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ The license key for the current software has been verified as ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ belonging to: ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Chai Research Corp. ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Shutdown handler not registered because Python interpreter is not running in the main thread
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
run pipeline %s
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
run pipeline stage %s
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Running pipeline stage MKMLizer
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Starting job with name chaiml-mistral31-24b-s-69496-v24-mkmlizer
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Waiting for job on chaiml-mistral31-24b-s-69496-v24-mkmlizer to finish
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Downloaded to shared memory in 44.064s
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Checking if ChaiML/nitral-dpos1-beta06-exp1-10kpref-1800-lowlr-2e-2-1 already exists in ChaiML
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmpooxfu2hy, device:0
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Version: 0.29.15 ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ https://mk1.ai ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ belonging to: ║
Shutdown handler not registered because Python interpreter is not running in the main thread
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
run pipeline %s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Chai Research Corp. ║
run pipeline stage %s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
Running pipeline stage MKMLizer
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
Starting job with name chaiml-slerpv5-mistral-24540-v28-mkmlizer
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Waiting for job on chaiml-slerpv5-mistral-24540-v28-mkmlizer to finish
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Version: 0.29.15 ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ https://mk1.ai ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ belonging to: ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Chai Research Corp. ║
chaiml-mistral32-deeps-89261-v11-mkmlizer: Downloaded to shared memory in 55.110s
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-mistral32-deeps-89261-v11-mkmlizer: Checking if ChaiML/mistral32_deepseek_cpos3_cpos21322_sfts11306_seq96_exp6 already exists in ChaiML
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-mistral32-deeps-89261-v11-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmpi8v_yyxi, device:0
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-mistral32-deeps-89261-v11-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: Downloaded to shared memory in 110.274s
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: Checking if ChaiML/cogito32b-dpos2-dpos1ft-b06-2e already exists in ChaiML
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmpisle3d3p, device:0
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Downloaded to shared memory in 80.368s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Checking if ChaiML/mistral31-24b-simpoexp1-s1-new-sft-retryv2top20lex-2e already exists in ChaiML
chaiml-mistral31-24b-s-69496-v24-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmpfv82i4tl, device:0
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-slerpv5-mistral-24540-v28-mkmlizer: Downloaded to shared memory in 73.625s
chaiml-slerpv5-mistral-24540-v28-mkmlizer: Checking if ChaiML/slerpv5_mistral31_simpo_dpo_dpoold_s2 already exists in ChaiML
chaiml-slerpv5-mistral-24540-v28-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmp3ybje2zs, device:0
chaiml-slerpv5-mistral-24540-v28-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: quantized model in 259.199s
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Processed model ChaiML/nitral-dpos1-beta06-exp1-10kpref-1800-lowlr-2e-2-1 in 303.264s
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nitral-dpos1-beta-5231-v7/nvidia
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nitral-dpos1-beta-5231-v7/nvidia/config.json
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nitral-dpos1-beta-5231-v7/nvidia/special_tokens_map.json
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nitral-dpos1-beta-5231-v7/nvidia/flywheel_model.0.safetensors
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 3/363 [00:01<03:22, 1.78it/s]
Loading 0: 1%| | 4/363 [00:03<05:34, 1.07it/s]
Loading 0: 1%|▏ | 5/363 [00:05<07:13, 1.21s/it]
Loading 0: 2%|▏ | 7/363 [00:05<03:56, 1.51it/s]
Loading 0: 2%|▏ | 8/363 [00:05<03:38, 1.62it/s]
Loading 0: 2%|▏ | 9/363 [00:06<03:24, 1.73it/s]
Loading 0: 3%|▎ | 10/363 [00:06<02:45, 2.14it/s]
Loading 0: 3%|▎ | 12/363 [00:08<03:40, 1.59it/s]
Loading 0: 4%|▎ | 13/363 [00:09<05:07, 1.14it/s]
Loading 0: 4%|▍ | 14/363 [00:11<06:28, 1.11s/it]
Loading 0: 4%|▍ | 16/363 [00:11<03:51, 1.50it/s]
Loading 0: 5%|▍ | 17/363 [00:12<03:34, 1.61it/s]
Loading 0: 5%|▍ | 18/363 [00:12<03:21, 1.72it/s]
Loading 0: 5%|▌ | 19/363 [00:12<02:45, 2.08it/s]
Loading 0: 6%|▌ | 21/363 [00:14<03:34, 1.60it/s]
Loading 0: 6%|▌ | 22/363 [00:16<04:58, 1.14it/s]
Loading 0: 6%|▋ | 23/363 [00:17<06:16, 1.11s/it]
Loading 0: 7%|▋ | 25/363 [00:18<03:46, 1.49it/s]
Loading 0: 7%|▋ | 26/363 [00:18<03:31, 1.60it/s]
Loading 0: 7%|▋ | 27/363 [00:19<03:19, 1.68it/s]
Loading 0: 8%|▊ | 28/363 [00:19<02:43, 2.05it/s]
Loading 0: 8%|▊ | 30/363 [00:20<03:31, 1.58it/s]
Loading 0: 9%|▊ | 31/363 [00:22<04:50, 1.14it/s]
Loading 0: 9%|▉ | 32/363 [00:24<06:05, 1.10s/it]
Loading 0: 9%|▉ | 34/363 [00:24<03:39, 1.50it/s]
Loading 0: 10%|▉ | 35/363 [00:24<03:24, 1.60it/s]
Loading 0: 10%|▉ | 36/363 [00:25<03:12, 1.70it/s]
Loading 0: 10%|█ | 37/363 [00:25<02:37, 2.07it/s]
Loading 0: 11%|█ | 39/363 [00:27<03:24, 1.58it/s]
Loading 0: 11%|█ | 40/363 [00:28<04:42, 1.14it/s]
Loading 0: 11%|█▏ | 41/363 [00:30<05:50, 1.09s/it]
Loading 0: 12%|█▏ | 43/363 [00:30<03:30, 1.52it/s]
Loading 0: 12%|█▏ | 44/363 [00:31<03:17, 1.62it/s]
Loading 0: 12%|█▏ | 45/363 [00:31<03:06, 1.70it/s]
Loading 0: 13%|█▎ | 46/363 [00:31<02:32, 2.07it/s]
Loading 0: 13%|█▎ | 48/363 [00:33<03:17, 1.59it/s]
Loading 0: 13%|█▎ | 49/363 [00:35<04:32, 1.15it/s]
Loading 0: 14%|█▍ | 50/363 [00:37<05:41, 1.09s/it]
Loading 0: 14%|█▍ | 52/363 [00:37<03:24, 1.52it/s]
Loading 0: 15%|█▍ | 53/363 [00:37<03:11, 1.62it/s]
Loading 0: 15%|█▍ | 54/363 [00:38<03:00, 1.72it/s]
Loading 0: 15%|█▌ | 55/363 [00:38<02:26, 2.11it/s]
Loading 0: 15%|█▌ | 56/363 [00:39<04:04, 1.26it/s]
Loading 0: 16%|█▌ | 57/363 [00:40<03:05, 1.65it/s]
Loading 0: 16%|█▌ | 58/363 [00:40<02:51, 1.77it/s]
Loading 0: 16%|█▋ | 59/363 [00:40<02:44, 1.85it/s]
Loading 0: 17%|█▋ | 60/363 [00:41<02:13, 2.27it/s]
Loading 0: 17%|█▋ | 61/363 [00:41<01:52, 2.70it/s]
Loading 0: 17%|█▋ | 62/363 [00:43<03:46, 1.33it/s]
Loading 0: 17%|█▋ | 63/363 [00:44<05:08, 1.03s/it]
Loading 0: 18%|█▊ | 64/363 [00:46<06:12, 1.24s/it]
Loading 0: 18%|█▊ | 66/363 [00:46<03:26, 1.44it/s]
Loading 0: 18%|█▊ | 67/363 [00:47<03:09, 1.57it/s]
Loading 0: 19%|█▊ | 68/363 [00:47<02:56, 1.67it/s]
Loading 0: 19%|█▉ | 69/363 [00:47<02:23, 2.06it/s]
Loading 0: 20%|█▉ | 71/363 [00:49<03:08, 1.55it/s]
Loading 0: 20%|█▉ | 72/363 [00:51<04:20, 1.12it/s]
Loading 0: 20%|██ | 73/363 [00:52<05:24, 1.12s/it]
Loading 0: 21%|██ | 75/363 [00:52<03:13, 1.49it/s]
Loading 0: 21%|██ | 76/363 [00:53<02:59, 1.60it/s]
Loading 0: 21%|██ | 77/363 [00:53<02:47, 1.70it/s]
Loading 0: 21%|██▏ | 78/363 [00:54<02:15, 2.10it/s]
Loading 0: 22%|██▏ | 80/363 [00:55<02:59, 1.58it/s]
Loading 0: 22%|██▏ | 81/363 [00:57<04:11, 1.12it/s]
Loading 0: 23%|██▎ | 82/363 [00:59<05:15, 1.12s/it]
Loading 0: 23%|██▎ | 84/363 [00:59<03:08, 1.48it/s]
Loading 0: 23%|██▎ | 85/363 [00:59<02:56, 1.57it/s]
Loading 0: 24%|██▎ | 86/363 [01:00<02:46, 1.66it/s]
Loading 0: 24%|██▍ | 87/363 [01:00<02:17, 2.01it/s]
Loading 0: 25%|██▍ | 89/363 [01:02<02:55, 1.56it/s]
Loading 0: 25%|██▍ | 90/363 [01:04<04:07, 1.10it/s]
Loading 0: 25%|██▌ | 91/363 [01:05<05:11, 1.15s/it]
Loading 0: 26%|██▌ | 93/363 [01:06<03:07, 1.44it/s]
Loading 0: 26%|██▌ | 94/363 [01:06<02:55, 1.53it/s]
Loading 0: 26%|██▌ | 95/363 [01:07<02:44, 1.63it/s]
Loading 0: 26%|██▋ | 96/363 [01:07<02:14, 1.98it/s]
Loading 0: 27%|██▋ | 98/363 [01:08<02:53, 1.53it/s]
Loading 0: 27%|██▋ | 99/363 [01:10<03:57, 1.11it/s]
Loading 0: 28%|██▊ | 100/363 [01:12<04:53, 1.12s/it]
Loading 0: 28%|██▊ | 102/363 [01:12<02:55, 1.49it/s]
Loading 0: 28%|██▊ | 103/363 [01:13<02:43, 1.59it/s]
Loading 0: 29%|██▊ | 104/363 [01:13<02:32, 1.69it/s]
Loading 0: 29%|██▉ | 105/363 [01:13<02:03, 2.09it/s]
Loading 0: 29%|██▉ | 106/363 [01:15<03:24, 1.25it/s]
Loading 0: 29%|██▉ | 107/363 [01:15<02:34, 1.65it/s]
Loading 0: 30%|██▉ | 108/363 [01:15<02:24, 1.76it/s]
Loading 0: 30%|███ | 109/363 [01:16<02:17, 1.85it/s]
Loading 0: 30%|███ | 110/363 [01:16<01:49, 2.31it/s]
Loading 0: 31%|███ | 112/363 [01:18<02:30, 1.67it/s]
Loading 0: 31%|███ | 113/363 [01:19<03:35, 1.16it/s]
Loading 0: 32%|███▏ | 116/363 [01:21<02:51, 1.44it/s]
Loading 0: 32%|███▏ | 117/363 [01:23<03:35, 1.14it/s]
Loading 0: 33%|███▎ | 118/363 [01:24<04:19, 1.06s/it]
Loading 0: 33%|███▎ | 120/363 [01:24<02:43, 1.49it/s]
Loading 0: 33%|███▎ | 121/363 [01:25<02:32, 1.59it/s]
Loading 0: 34%|███▎ | 122/363 [01:25<02:22, 1.69it/s]
Loading 0: 34%|███▍ | 123/363 [01:25<01:56, 2.05it/s]
Loading 0: 34%|███▍ | 125/363 [01:27<02:28, 1.61it/s]
Loading 0: 35%|███▍ | 126/363 [01:29<03:22, 1.17it/s]
Loading 0: 35%|███▍ | 127/363 [01:30<04:09, 1.06s/it]
Loading 0: 36%|███▌ | 129/363 [01:31<02:30, 1.56it/s]
Loading 0: 36%|███▌ | 130/363 [01:31<02:19, 1.67it/s]
Loading 0: 36%|███▌ | 131/363 [01:31<02:11, 1.77it/s]
Loading 0: 36%|███▋ | 132/363 [01:32<01:47, 2.16it/s]
Loading 0: 37%|███▋ | 134/363 [01:33<02:19, 1.64it/s]
Loading 0: 37%|███▋ | 135/363 [01:35<03:13, 1.18it/s]
Loading 0: 37%|███▋ | 136/363 [01:37<04:00, 1.06s/it]
Loading 0: 38%|███▊ | 138/363 [01:37<02:24, 1.56it/s]
Loading 0: 38%|███▊ | 139/363 [01:37<02:14, 1.66it/s]
Loading 0: 39%|███▊ | 140/363 [01:38<02:06, 1.76it/s]
Loading 0: 39%|███▉ | 141/363 [01:38<01:42, 2.16it/s]
Loading 0: 39%|███▉ | 142/363 [01:38<01:24, 2.62it/s]
Loading 0: 39%|███▉ | 143/363 [01:40<02:40, 1.37it/s]
Loading 0: 40%|███▉ | 144/363 [01:41<03:41, 1.01s/it]
Loading 0: 40%|████ | 147/363 [01:43<02:41, 1.34it/s]
Loading 0: 41%|████ | 148/363 [01:45<03:20, 1.07it/s]
Loading 0: 41%|████ | 149/363 [01:46<03:58, 1.11s/it]
Loading 0: 42%|████▏ | 151/363 [01:46<02:27, 1.44it/s]
Loading 0: 42%|████▏ | 152/363 [01:47<02:15, 1.56it/s]
Loading 0: 42%|████▏ | 153/363 [01:47<02:06, 1.66it/s]
Loading 0: 42%|████▏ | 154/363 [01:47<01:43, 2.02it/s]
Loading 0: 43%|████▎ | 156/363 [01:49<02:10, 1.59it/s]
Loading 0: 43%|████▎ | 157/363 [01:51<02:57, 1.16it/s]
Loading 0: 44%|████▎ | 158/363 [01:52<03:39, 1.07s/it]
Loading 0: 44%|████▍ | 160/363 [01:53<02:11, 1.54it/s]
Loading 0: 44%|████▍ | 161/363 [01:53<02:01, 1.66it/s]
Loading 0: 45%|████▍ | 162/363 [01:53<01:54, 1.75it/s]
Loading 0: 45%|████▍ | 163/363 [01:54<01:33, 2.15it/s]
Loading 0: 45%|████▌ | 165/363 [01:55<02:00, 1.64it/s]
Loading 0: 46%|████▌ | 166/363 [01:57<02:47, 1.18it/s]
Loading 0: 46%|████▌ | 167/363 [01:59<03:29, 1.07s/it]
Loading 0: 47%|████▋ | 169/363 [01:59<02:05, 1.55it/s]
Loading 0: 47%|████▋ | 170/363 [01:59<01:55, 1.67it/s]
Loading 0: 47%|████▋ | 171/363 [02:00<01:48, 1.76it/s]
Loading 0: 47%|████▋ | 172/363 [02:00<01:28, 2.16it/s]
Loading 0: 48%|████▊ | 174/363 [02:01<01:55, 1.64it/s]
Loading 0: 48%|████▊ | 175/363 [02:03<02:38, 1.18it/s]
Loading 0: 48%|████▊ | 176/363 [02:05<03:17, 1.06s/it]
Loading 0: 49%|████▉ | 178/363 [02:05<01:57, 1.57it/s]
Loading 0: 49%|████▉ | 179/363 [02:05<01:49, 1.68it/s]
Loading 0: 50%|████▉ | 180/363 [02:06<01:43, 1.77it/s]
Loading 0: 50%|████▉ | 181/363 [02:06<01:24, 2.15it/s]
Loading 0: 50%|█████ | 183/363 [02:07<01:49, 1.65it/s]
Loading 0: 51%|█████ | 184/363 [02:09<02:31, 1.18it/s]
Loading 0: 51%|█████ | 185/363 [02:11<03:06, 1.05s/it]
Loading 0: 52%|█████▏ | 187/363 [02:11<01:51, 1.58it/s]
Loading 0: 52%|█████▏ | 188/363 [02:11<01:43, 1.69it/s]
Loading 0: 52%|█████▏ | 189/363 [02:12<01:37, 1.78it/s]
Loading 0: 52%|█████▏ | 190/363 [02:12<01:20, 2.15it/s]
Loading 0: 53%|█████▎ | 192/363 [02:14<01:43, 1.65it/s]
Loading 0: 53%|█████▎ | 193/363 [02:15<02:23, 1.19it/s]
Loading 0: 53%|█████▎ | 194/363 [02:17<02:58, 1.06s/it]
Loading 0: 54%|█████▍ | 196/363 [02:17<01:46, 1.57it/s]
Loading 0: 54%|█████▍ | 197/363 [02:17<01:38, 1.69it/s]
Loading 0: 55%|█████▍ | 198/363 [02:18<01:32, 1.78it/s]
Loading 0: 55%|█████▍ | 199/363 [02:18<01:15, 2.18it/s]
Loading 0: 55%|█████▌ | 201/363 [02:20<01:38, 1.65it/s]
Loading 0: 56%|█████▌ | 202/363 [02:21<02:15, 1.19it/s]
Loading 0: 56%|█████▌ | 203/363 [02:23<02:48, 1.05s/it]
Loading 0: 56%|█████▋ | 205/363 [02:23<01:40, 1.57it/s]
Loading 0: 57%|█████▋ | 206/363 [02:24<01:33, 1.68it/s]
Loading 0: 57%|█████▋ | 207/363 [02:24<01:27, 1.79it/s]
Loading 0: 57%|█████▋ | 208/363 [02:24<01:10, 2.19it/s]
Loading 0: 58%|█████▊ | 210/363 [02:26<01:32, 1.65it/s]
Loading 0: 58%|█████▊ | 211/363 [02:27<02:08, 1.18it/s]
Loading 0: 58%|█████▊ | 212/363 [02:29<02:38, 1.05s/it]
Loading 0: 59%|█████▉ | 214/363 [02:29<01:34, 1.57it/s]
Loading 0: 59%|█████▉ | 215/363 [02:30<01:27, 1.69it/s]
Loading 0: 60%|█████▉ | 216/363 [02:30<01:22, 1.78it/s]
Loading 0: 60%|█████▉ | 217/363 [02:30<01:07, 2.16it/s]
Loading 0: 60%|██████ | 218/363 [02:32<01:53, 1.28it/s]
Loading 0: 60%|██████ | 219/363 [02:32<01:25, 1.69it/s]
Loading 0: 61%|██████ | 220/363 [02:32<01:18, 1.82it/s]
Loading 0: 61%|██████ | 221/363 [02:33<01:14, 1.90it/s]
Loading 0: 61%|██████ | 222/363 [02:33<00:59, 2.37it/s]
Loading 0: 61%|██████▏ | 223/363 [02:33<00:49, 2.85it/s]
Loading 0: 62%|██████▏ | 224/363 [02:35<01:40, 1.38it/s]
Loading 0: 62%|██████▏ | 225/363 [02:37<02:19, 1.01s/it]
Loading 0: 63%|██████▎ | 228/363 [02:38<01:41, 1.33it/s]
Loading 0: 63%|██████▎ | 229/363 [02:40<02:04, 1.07it/s]
Loading 0: 63%|██████▎ | 230/363 [02:42<02:27, 1.11s/it]
Loading 0: 64%|██████▍ | 232/363 [02:42<01:30, 1.45it/s]
Loading 0: 64%|██████▍ | 233/363 [02:42<01:22, 1.57it/s]
Loading 0: 64%|██████▍ | 234/363 [02:43<01:16, 1.68it/s]
Loading 0: 65%|██████▍ | 235/363 [02:43<01:02, 2.06it/s]
Loading 0: 65%|██████▌ | 237/363 [02:44<01:18, 1.61it/s]
Loading 0: 66%|██████▌ | 238/363 [02:46<01:46, 1.17it/s]
Loading 0: 66%|██████▌ | 239/363 [02:48<02:11, 1.06s/it]
Loading 0: 66%|██████▋ | 241/363 [02:48<01:18, 1.55it/s]
Loading 0: 67%|██████▋ | 242/363 [02:48<01:12, 1.67it/s]
Loading 0: 67%|██████▋ | 243/363 [02:49<01:07, 1.78it/s]
Loading 0: 67%|██████▋ | 244/363 [02:49<00:55, 2.16it/s]
Loading 0: 68%|██████▊ | 246/363 [02:50<01:11, 1.64it/s]
Loading 0: 68%|██████▊ | 247/363 [02:52<01:38, 1.18it/s]
Loading 0: 68%|██████▊ | 248/363 [02:54<02:01, 1.05s/it]
Loading 0: 69%|██████▉ | 250/363 [02:54<01:12, 1.57it/s]
Loading 0: 69%|██████▉ | 251/363 [02:54<01:06, 1.68it/s]
Loading 0: 69%|██████▉ | 252/363 [02:55<01:02, 1.78it/s]
Loading 0: 70%|██████▉ | 253/363 [02:55<00:50, 2.18it/s]
Loading 0: 70%|███████ | 255/363 [02:57<01:05, 1.66it/s]
Loading 0: 71%|███████ | 256/363 [02:58<01:30, 1.19it/s]
Loading 0: 71%|███████ | 257/363 [03:00<01:51, 1.05s/it]
Loading 0: 71%|███████▏ | 259/363 [03:00<01:06, 1.57it/s]
Loading 0: 72%|███████▏ | 260/363 [03:00<01:01, 1.67it/s]
Loading 0: 72%|███████▏ | 261/363 [03:01<00:57, 1.77it/s]
Loading 0: 72%|███████▏ | 262/363 [03:01<00:46, 2.17it/s]
Loading 0: 73%|███████▎ | 264/363 [03:03<01:00, 1.65it/s]
Loading 0: 73%|███████▎ | 265/363 [03:04<01:22, 1.18it/s]
Loading 0: 73%|███████▎ | 266/363 [03:06<01:42, 1.05s/it]
Loading 0: 74%|███████▍ | 268/363 [03:06<01:00, 1.57it/s]
Loading 0: 74%|███████▍ | 269/363 [03:07<00:55, 1.69it/s]
Loading 0: 74%|███████▍ | 270/363 [03:07<00:52, 1.78it/s]
Loading 0: 75%|███████▍ | 271/363 [03:07<00:42, 2.18it/s]
Loading 0: 75%|███████▌ | 273/363 [03:09<00:54, 1.65it/s]
Loading 0: 75%|███████▌ | 274/363 [03:10<01:14, 1.19it/s]
Loading 0: 76%|███████▌ | 275/363 [03:12<01:32, 1.05s/it]
Loading 0: 76%|███████▋ | 277/363 [03:12<00:54, 1.57it/s]
Loading 0: 77%|███████▋ | 278/363 [03:13<00:50, 1.69it/s]
Loading 0: 77%|███████▋ | 279/363 [03:13<00:47, 1.78it/s]
Loading 0: 77%|███████▋ | 280/363 [03:13<00:38, 2.18it/s]
Loading 0: 78%|███████▊ | 282/363 [03:15<00:49, 1.65it/s]
Loading 0: 78%|███████▊ | 283/363 [03:16<01:07, 1.19it/s]
Loading 0: 78%|███████▊ | 284/363 [03:18<01:23, 1.06s/it]
Loading 0: 79%|███████▉ | 286/363 [03:18<00:49, 1.57it/s]
Loading 0: 79%|███████▉ | 287/363 [03:19<00:45, 1.67it/s]
Loading 0: 79%|███████▉ | 288/363 [03:19<00:42, 1.77it/s]
Loading 0: 80%|███████▉ | 289/363 [03:19<00:34, 2.17it/s]
Loading 0: 80%|████████ | 291/363 [03:21<00:43, 1.64it/s]
Loading 0: 80%|████████ | 292/363 [03:23<01:00, 1.18it/s]
Loading 0: 81%|████████ | 293/363 [03:24<01:13, 1.06s/it]
Loading 0: 81%|████████▏ | 295/363 [03:24<00:43, 1.57it/s]
Loading 0: 82%|████████▏ | 296/363 [03:25<00:39, 1.69it/s]
Loading 0: 82%|████████▏ | 297/363 [03:25<00:37, 1.78it/s]
Loading 0: 82%|████████▏ | 298/363 [03:26<00:30, 2.16it/s]
Loading 0: 82%|████████▏ | 299/363 [03:27<00:49, 1.29it/s]
Loading 0: 83%|████████▎ | 301/363 [03:28<00:34, 1.81it/s]
Loading 0: 83%|████████▎ | 302/363 [03:28<00:32, 1.88it/s]
Loading 0: 83%|████████▎ | 303/363 [03:28<00:26, 2.28it/s]
Loading 0: 84%|████████▎ | 304/363 [03:29<00:24, 2.45it/s]
Loading 0: 84%|████████▍ | 306/363 [03:30<00:32, 1.73it/s]
Loading 0: 85%|████████▍ | 307/363 [03:32<00:46, 1.20it/s]
Loading 0: 85%|████████▌ | 310/363 [03:33<00:36, 1.46it/s]
Loading 0: 86%|████████▌ | 311/363 [03:35<00:45, 1.16it/s]
Loading 0: 86%|████████▌ | 312/363 [03:37<00:53, 1.05s/it]
Loading 0: 87%|████████▋ | 314/363 [03:37<00:32, 1.50it/s]
Loading 0: 87%|████████▋ | 315/363 [03:37<00:29, 1.61it/s]
Loading 0: 87%|████████▋ | 316/363 [03:38<00:27, 1.71it/s]
Loading 0: 87%|████████▋ | 317/363 [03:38<00:21, 2.09it/s]
Loading 0: 88%|████████▊ | 319/363 [03:40<00:26, 1.63it/s]
Loading 0: 88%|████████▊ | 320/363 [03:41<00:36, 1.19it/s]
Loading 0: 88%|████████▊ | 321/363 [03:43<00:44, 1.05s/it]
Loading 0: 89%|████████▉ | 323/363 [03:43<00:25, 1.57it/s]
Loading 0: 89%|████████▉ | 324/363 [03:43<00:23, 1.68it/s]
Loading 0: 90%|████████▉ | 325/363 [03:44<00:21, 1.78it/s]
Loading 0: 90%|████████▉ | 326/363 [03:44<00:16, 2.18it/s]
Loading 0: 90%|█████████ | 328/363 [03:46<00:21, 1.66it/s]
Loading 0: 91%|█████████ | 329/363 [03:47<00:28, 1.20it/s]
Loading 0: 91%|█████████ | 330/363 [03:49<00:34, 1.04s/it]
Loading 0: 91%|█████████▏| 332/363 [03:49<00:19, 1.58it/s]
Loading 0: 92%|█████████▏| 333/363 [03:49<00:17, 1.69it/s]
Loading 0: 92%|█████████▏| 334/363 [03:50<00:16, 1.78it/s]
Loading 0: 92%|█████████▏| 335/363 [03:50<00:12, 2.18it/s]
Loading 0: 93%|█████████▎| 337/363 [03:52<00:15, 1.66it/s]
Loading 0: 93%|█████████▎| 338/363 [03:53<00:21, 1.19it/s]
Loading 0: 93%|█████████▎| 339/363 [03:55<00:25, 1.05s/it]
Loading 0: 94%|█████████▍| 341/363 [03:55<00:13, 1.57it/s]
Loading 0: 94%|█████████▍| 342/363 [03:56<00:12, 1.69it/s]
Loading 0: 94%|█████████▍| 343/363 [03:56<00:11, 1.79it/s]
Loading 0: 95%|█████████▍| 344/363 [03:56<00:08, 2.18it/s]
Loading 0: 95%|█████████▌| 346/363 [03:58<00:10, 1.65it/s]
Loading 0: 96%|█████████▌| 347/363 [03:59<00:13, 1.19it/s]
Loading 0: 96%|█████████▌| 348/363 [04:01<00:15, 1.05s/it]
Loading 0: 96%|█████████▋| 350/363 [04:01<00:08, 1.58it/s]
Loading 0: 97%|█████████▋| 351/363 [04:02<00:07, 1.70it/s]
Loading 0: 97%|█████████▋| 352/363 [04:02<00:06, 1.79it/s]
Loading 0: 97%|█████████▋| 353/363 [04:02<00:04, 2.19it/s]
Loading 0: 98%|█████████▊| 355/363 [04:04<00:04, 1.65it/s]
Loading 0: 98%|█████████▊| 356/363 [04:06<00:05, 1.19it/s]
Loading 0: 98%|█████████▊| 357/363 [04:07<00:06, 1.05s/it]
Loading 0: 99%|█████████▉| 359/363 [04:07<00:02, 1.57it/s]
Loading 0: 99%|█████████▉| 360/363 [04:08<00:01, 1.68it/s]
Loading 0: 99%|█████████▉| 361/363 [04:08<00:01, 1.79it/s]
Loading 0: 100%|█████████▉| 362/363 [04:08<00:00, 2.19it/s]
Job chaiml-nitral-dpos1-beta-5231-v7-mkmlizer completed after 331.02s with status: succeeded
Stopping job with name chaiml-nitral-dpos1-beta-5231-v7-mkmlizer
Pipeline stage MKMLizer completed in 333.08s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.77s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nitral-dpos1-beta-5231-v7
Waiting for inference service chaiml-nitral-dpos1-beta-5231-v7 to be ready
chaiml-cogito32b-dpos2-39363-v8-mkmlizer:
Loading 0: 0%| | 0/771 [00:00<?, ?it/s]
Loading 0: 0%| | 3/771 [00:03<13:14, 1.03s/it]
Loading 0: 1%| | 4/771 [00:06<21:51, 1.71s/it]
Loading 0: 1%| | 5/771 [00:09<27:51, 2.18s/it]
Loading 0: 1%| | 8/771 [00:09<12:04, 1.05it/s]
Loading 0: 1%|▏ | 10/771 [00:10<09:06, 1.39it/s]
Loading 0: 1%|▏ | 11/771 [00:10<08:45, 1.45it/s]
Loading 0: 2%|▏ | 13/771 [00:10<06:07, 2.06it/s]
Loading 0: 2%|▏ | 15/771 [00:14<10:33, 1.19it/s]
Loading 0: 2%|▏ | 16/771 [00:16<14:24, 1.15s/it]
Loading 0: 2%|▏ | 17/771 [00:18<18:10, 1.45s/it]
Loading 0: 3%|▎ | 20/771 [00:19<09:32, 1.31it/s]
Loading 0: 3%|▎ | 22/771 [00:19<07:40, 1.63it/s]
Loading 0: 3%|▎ | 23/771 [00:20<07:31, 1.66it/s]
Loading 0: 3%|▎ | 25/771 [00:20<05:25, 2.29it/s]
Loading 0: 4%|▎ | 27/771 [00:23<09:59, 1.24it/s]
Loading 0: 4%|▎ | 28/771 [00:25<13:39, 1.10s/it]
Loading 0: 4%|▍ | 29/771 [00:28<17:18, 1.40s/it]
Loading 0: 4%|▍ | 32/771 [00:28<09:12, 1.34it/s]
Loading 0: 4%|▍ | 34/771 [00:29<07:26, 1.65it/s]
Loading 0: 5%|▍ | 35/771 [00:29<07:20, 1.67it/s]
Loading 0: 5%|▍ | 37/771 [00:29<05:20, 2.29it/s]
Loading 0: 5%|▍ | 38/771 [00:32<10:37, 1.15it/s]
Loading 0: 5%|▌ | 40/771 [00:32<07:02, 1.73it/s]
Loading 0: 5%|▌ | 41/771 [00:33<07:01, 1.73it/s]
Loading 0: 6%|▌ | 43/771 [00:33<05:42, 2.13it/s]
Loading 0: 6%|▌ | 45/771 [00:34<04:17, 2.82it/s]
Loading 0: 6%|▌ | 46/771 [00:34<03:51, 3.13it/s]
Loading 0: 6%|▌ | 47/771 [00:37<11:09, 1.08it/s]
Loading 0: 6%|▌ | 48/771 [00:40<16:21, 1.36s/it]
Loading 0: 7%|▋ | 51/771 [00:43<14:14, 1.19s/it]
Loading 0: 7%|▋ | 52/771 [00:46<18:14, 1.52s/it]
Loading 0: 7%|▋ | 53/771 [00:48<21:58, 1.84s/it]
Loading 0: 7%|▋ | 56/771 [00:49<11:33, 1.03it/s]
Loading 0: 7%|▋ | 57/771 [00:49<10:40, 1.11it/s]
Loading 0: 8%|▊ | 59/771 [00:50<08:07, 1.46it/s]
Loading 0: 8%|▊ | 61/771 [00:50<05:55, 2.00it/s]
Loading 0: 8%|▊ | 63/771 [00:53<09:50, 1.20it/s]
Loading 0: 8%|▊ | 64/771 [00:56<14:42, 1.25s/it]
Loading 0: 8%|▊ | 65/771 [00:59<19:20, 1.64s/it]
Loading 0: 9%|▉ | 68/771 [00:59<10:19, 1.13it/s]
Loading 0: 9%|▉ | 69/771 [01:00<09:38, 1.21it/s]
Loading 0: 9%|▉ | 71/771 [01:01<07:27, 1.57it/s]
Loading 0: 9%|▉ | 73/771 [01:01<05:26, 2.14it/s]
Loading 0: 10%|▉ | 75/771 [01:04<09:25, 1.23it/s]
Loading 0: 10%|▉ | 76/771 [01:07<14:16, 1.23s/it]
Loading 0: 10%|▉ | 77/771 [01:10<18:57, 1.64s/it]
Loading 0: 10%|█ | 80/771 [01:10<10:08, 1.13it/s]
Loading 0: 11%|█ | 81/771 [01:11<09:29, 1.21it/s]
Loading 0: 11%|█ | 83/771 [01:11<07:19, 1.56it/s]
Loading 0: 11%|█ | 85/771 [01:12<05:22, 2.12it/s]
Loading 0: 11%|█▏ | 87/771 [01:15<09:15, 1.23it/s]
Loading 0: 11%|█▏ | 88/771 [01:18<14:07, 1.24s/it]
Loading 0: 12%|█▏ | 89/771 [01:21<18:42, 1.65s/it]
Loading 0: 12%|█▏ | 92/771 [01:21<10:00, 1.13it/s]
Loading 0: 12%|█▏ | 93/771 [01:22<09:20, 1.21it/s]
Loading 0: 12%|█▏ | 95/771 [01:22<07:12, 1.56it/s]
Loading 0: 13%|█▎ | 97/771 [01:22<05:18, 2.12it/s]
Loading 0: 13%|█▎ | 98/771 [01:26<10:51, 1.03it/s]
Loading 0: 13%|█▎ | 100/771 [01:26<07:14, 1.55it/s]
Loading 0: 13%|█▎ | 101/771 [01:26<07:03, 1.58it/s]
Loading 0: 13%|█▎ | 103/771 [01:27<05:38, 1.98it/s]
Loading 0: 14%|█▎ | 105/771 [01:27<04:12, 2.64it/s]
Loading 0: 14%|█▎ | 106/771 [01:27<03:44, 2.96it/s]
Loading 0: 14%|█▍ | 107/771 [01:30<10:17, 1.07it/s]
Loading 0: 14%|█▍ | 108/771 [01:33<15:54, 1.44s/it]
Loading 0: 14%|█▍ | 109/771 [01:37<20:38, 1.87s/it]
Loading 0: 15%|█▍ | 112/771 [01:37<10:03, 1.09it/s]
Loading 0: 15%|█▍ | 113/771 [01:37<09:17, 1.18it/s]
Loading 0: 15%|█▍ | 115/771 [01:38<07:02, 1.55it/s]
Loading 0: 15%|█▌ | 117/771 [01:38<05:05, 2.14it/s]
Loading 0: 15%|█▌ | 119/771 [01:41<08:51, 1.23it/s]
Loading 0: 16%|█▌ | 120/771 [01:44<13:28, 1.24s/it]
Loading 0: 16%|█▌ | 121/771 [01:47<17:52, 1.65s/it]
Loading 0: 16%|█▌ | 124/771 [01:47<09:28, 1.14it/s]
Loading 0: 16%|█▋ | 126/771 [01:48<07:27, 1.44it/s]
Loading 0: 16%|█▋ | 127/771 [01:49<07:12, 1.49it/s]
Loading 0: 17%|█▋ | 129/771 [01:49<05:09, 2.07it/s]
Loading 0: 17%|█▋ | 131/771 [01:52<08:49, 1.21it/s]
Loading 0: 17%|█▋ | 132/771 [01:55<13:18, 1.25s/it]
Loading 0: 17%|█▋ | 133/771 [01:58<17:36, 1.66s/it]
Loading 0: 18%|█▊ | 137/771 [01:59<08:50, 1.19it/s]
Loading 0: 18%|█▊ | 139/771 [01:59<07:11, 1.46it/s]
Loading 0: 18%|█▊ | 141/771 [02:00<05:29, 1.91it/s]
Loading 0: 18%|█▊ | 142/771 [02:03<09:59, 1.05it/s]
Loading 0: 19%|█▊ | 144/771 [02:03<06:53, 1.52it/s]
Loading 0: 19%|█▉ | 145/771 [02:03<06:42, 1.55it/s]
Loading 0: 19%|█▉ | 147/771 [02:04<05:23, 1.93it/s]
Loading 0: 19%|█▉ | 149/771 [02:04<04:02, 2.57it/s]
Loading 0: 20%|█▉ | 151/771 [02:07<07:47, 1.33it/s]
Loading 0: 20%|█▉ | 152/771 [02:10<12:24, 1.20s/it]
Loading 0: 20%|██ | 155/771 [02:13<11:31, 1.12s/it]
Loading 0: 20%|██ | 156/771 [02:16<15:00, 1.46s/it]
Loading 0: 20%|██ | 157/771 [02:20<18:25, 1.80s/it]
Loading 0: 21%|██ | 160/771 [02:20<10:03, 1.01it/s]
Loading 0: 21%|██ | 161/771 [02:20<09:16, 1.10it/s]
Loading 0: 21%|██ | 163/771 [02:21<07:05, 1.43it/s]
Loading 0: 21%|██▏ | 165/771 [02:21<05:11, 1.95it/s]
Loading 0: 22%|██▏ | 166/771 [02:21<04:31, 2.23it/s]
Loading 0: 22%|██▏ | 167/771 [02:24<10:07, 1.01s/it]
Loading 0: 22%|██▏ | 168/771 [02:28<15:08, 1.51s/it]
Loading 0: 22%|██▏ | 171/771 [02:31<12:39, 1.27s/it]
Loading 0: 22%|██▏ | 172/771 [02:34<16:06, 1.61s/it]
Loading 0: 22%|██▏ | 173/771 [02:37<19:21, 1.94s/it]
Loading 0: 23%|██▎ | 176/771 [02:37<10:10, 1.03s/it]
Loading 0: 23%|██▎ | 177/771 [02:37<09:16, 1.07it/s]
Loading 0: 23%|██▎ | 179/771 [02:38<07:01, 1.40it/s]
Loading 0: 23%|██▎ | 181/771 [02:38<05:03, 1.94it/s]
Loading 0: 24%|██▎ | 183/771 [02:41<08:12, 1.19it/s]
Loading 0: 24%|██▍ | 184/771 [02:44<12:15, 1.25s/it]
Loading 0: 24%|██▍ | 185/771 [02:47<16:04, 1.65s/it]
Loading 0: 24%|██▍ | 188/771 [02:48<08:34, 1.13it/s]
Loading 0: 25%|██▍ | 190/771 [02:48<06:42, 1.44it/s]
Loading 0: 25%|██▍ | 191/771 [02:49<06:29, 1.49it/s]
Loading 0: 25%|██▌ | 193/771 [02:49<04:39, 2.07it/s]
Loading 0: 25%|██▌ | 195/771 [02:52<07:55, 1.21it/s]
Loading 0: 25%|██▌ | 196/771 [02:55<11:59, 1.25s/it]
Loading 0: 26%|██▌ | 197/771 [02:58<15:48, 1.65s/it]
Loading 0: 26%|██▌ | 200/771 [02:58<08:23, 1.13it/s]
Loading 0: 26%|██▌ | 201/771 [02:59<07:49, 1.21it/s]
Loading 0: 26%|██▋ | 203/771 [02:59<06:02, 1.57it/s]
Loading 0: 27%|██▋ | 205/771 [03:00<04:25, 2.13it/s]
Loading 0: 27%|██▋ | 207/771 [03:03<07:36, 1.24it/s]
Loading 0: 27%|██▋ | 208/771 [03:06<11:46, 1.25s/it]
Loading 0: 27%|██▋ | 209/771 [03:09<15:39, 1.67s/it]
Loading 0: 27%|██▋ | 212/771 [03:09<08:21, 1.11it/s]
Loading 0: 28%|██▊ | 213/771 [03:10<07:49, 1.19it/s]
Loading 0: 28%|██▊ | 215/771 [03:11<06:02, 1.53it/s]
Loading 0: 28%|██▊ | 217/771 [03:11<04:24, 2.10it/s]
Loading 0: 28%|██▊ | 218/771 [03:14<09:01, 1.02it/s]
Loading 0: 29%|██▊ | 220/771 [03:14<06:00, 1.53it/s]
Loading 0: 29%|██▊ | 221/771 [03:15<05:52, 1.56it/s]
Loading 0: 29%|██▉ | 223/771 [03:15<04:42, 1.94it/s]
Loading 0: 29%|██▉ | 225/771 [03:15<03:27, 2.63it/s]
Loading 0: 29%|██▉ | 226/771 [03:16<03:04, 2.95it/s]
Loading 0: 29%|██▉ | 227/771 [03:19<08:33, 1.06it/s]
Loading 0: 30%|██▉ | 228/771 [03:22<13:22, 1.48s/it]
Loading 0: 30%|██▉ | 231/771 [03:25<11:17, 1.26s/it]
Loading 0: 30%|███ | 232/771 [03:28<14:30, 1.61s/it]
Loading 0: 30%|███ | 233/771 [03:31<17:32, 1.96s/it]
Loading 0: 31%|███ | 236/771 [03:31<09:12, 1.03s/it]
Loading 0: 31%|███ | 237/771 [03:32<08:26, 1.05it/s]
Loading 0: 31%|███ | 239/771 [03:33<06:24, 1.38it/s]
Loading 0: 31%|███▏ | 241/771 [03:33<04:36, 1.92it/s]
Loading 0: 32%|███▏ | 243/771 [03:36<07:30, 1.17it/s]
Loading 0: 32%|███▏ | 244/771 [03:39<11:10, 1.27s/it]
Loading 0: 32%|███▏ | 245/771 [03:42<14:39, 1.67s/it]
Loading 0: 32%|███▏ | 248/771 [03:42<07:48, 1.12it/s]
Loading 0: 32%|███▏ | 250/771 [03:43<06:09, 1.41it/s]
Loading 0: 33%|███▎ | 251/771 [03:43<05:57, 1.46it/s]
Loading 0: 33%|███▎ | 253/771 [03:44<04:14, 2.03it/s]
Loading 0: 33%|███▎ | 255/771 [03:47<07:13, 1.19it/s]
Loading 0: 33%|███▎ | 256/771 [03:50<10:53, 1.27s/it]
Loading 0: 33%|███▎ | 257/771 [03:53<14:20, 1.67s/it]
Loading 0: 34%|███▍ | 261/771 [03:54<07:12, 1.18it/s]
Loading 0: 34%|███▍ | 263/771 [03:54<05:51, 1.44it/s]
Loading 0: 34%|███▍ | 265/771 [03:55<04:26, 1.90it/s]
Loading 0: 35%|███▍ | 267/771 [03:58<06:56, 1.21it/s]
Loading 0: 35%|███▍ | 268/771 [04:01<10:11, 1.22s/it]
Loading 0: 35%|███▍ | 269/771 [04:04<13:24, 1.60s/it]
Loading 0: 35%|███▌ | 272/771 [04:04<07:21, 1.13it/s]
Loading 0: 35%|███▌ | 273/771 [04:05<06:54, 1.20it/s]
Loading 0: 36%|███▌ | 275/771 [04:05<05:22, 1.54it/s]
Loading 0: 36%|███▌ | 277/771 [04:05<03:55, 2.10it/s]
Loading 0: 36%|███▌ | 278/771 [04:09<08:00, 1.03it/s]
Loading 0: 36%|███▋ | 280/771 [04:09<05:20, 1.53it/s]
Loading 0: 36%|███▋ | 281/771 [04:09<05:13, 1.56it/s]
Loading 0: 37%|███▋ | 283/771 [04:10<04:10, 1.95it/s]
Loading 0: 37%|███▋ | 285/771 [04:10<03:04, 2.63it/s]
Loading 0: 37%|███▋ | 286/771 [04:10<02:43, 2.96it/s]
Loading 0: 37%|███▋ | 287/771 [04:13<07:36, 1.06it/s]
Loading 0: 37%|███▋ | 288/771 [04:17<12:06, 1.50s/it]
Loading 0: 38%|███▊ | 291/771 [04:20<10:19, 1.29s/it]
Loading 0: 38%|███▊ | 292/771 [04:23<13:10, 1.65s/it]
Loading 0: 38%|███▊ | 293/771 [04:26<15:52, 1.99s/it]
Loading 0: 38%|███▊ | 296/771 [04:26<08:19, 1.05s/it]
Loading 0: 39%|███▊ | 297/771 [04:27<07:37, 1.04it/s]
Loading 0: 39%|███▉ | 299/771 [04:28<05:45, 1.37it/s]
Loading 0: 39%|███▉ | 301/771 [04:28<04:08, 1.89it/s]
Loading 0: 39%|███▉ | 303/771 [04:31<06:44, 1.16it/s]
Loading 0: 39%|███▉ | 304/771 [04:34<10:02, 1.29s/it]
Loading 0: 40%|███▉ | 305/771 [04:37<13:10, 1.70s/it]
Loading 0: 40%|███▉ | 308/771 [04:37<07:00, 1.10it/s]
Loading 0: 40%|████ | 310/771 [04:38<05:30, 1.39it/s]
Loading 0: 40%|████ | 311/771 [04:39<05:19, 1.44it/s]
Loading 0: 41%|████ | 313/771 [04:39<03:47, 2.02it/s]
Loading 0: 41%|████ | 315/771 [04:42<06:27, 1.18it/s]
Loading 0: 41%|████ | 316/771 [04:45<09:44, 1.28s/it]
Loading 0: 41%|████ | 317/771 [04:48<12:51, 1.70s/it]
Loading 0: 42%|████▏ | 320/771 [04:48<06:48, 1.10it/s]
Loading 0: 42%|████▏ | 321/771 [04:49<06:21, 1.18it/s]
Loading 0: 42%|████▏ | 323/771 [04:50<04:54, 1.52it/s]
Loading 0: 42%|████▏ | 325/771 [04:50<03:33, 2.09it/s]
Loading 0: 42%|████▏ | 327/771 [04:53<06:08, 1.20it/s]
Loading 0: 43%|████▎ | 328/771 [04:56<09:20, 1.26s/it]
Loading 0: 43%|████▎ | 329/771 [04:59<12:21, 1.68s/it]
Loading 0: 43%|████▎ | 332/771 [05:00<06:34, 1.11it/s]
Loading 0: 43%|████▎ | 334/771 [05:00<05:11, 1.40it/s]
Loading 0: 43%|████▎ | 335/771 [05:01<05:00, 1.45it/s]
Loading 0: 44%|████▎ | 337/771 [05:01<03:34, 2.02it/s]
Loading 0: 44%|████▍ | 338/771 [05:04<07:19, 1.01s/it]
Loading 0: 44%|████▍ | 340/771 [05:04<04:48, 1.49it/s]
Loading 0: 44%|████▍ | 341/771 [05:05<04:41, 1.53it/s]
Loading 0: 44%|████▍ | 343/771 [05:05<03:43, 1.92it/s]
Loading 0: 45%|████▍ | 345/771 [05:06<02:42, 2.62it/s]
Loading 0: 45%|████▍ | 346/771 [05:06<02:24, 2.94it/s]
Loading 0: 45%|████▌ | 347/771 [05:09<06:47, 1.04it/s]
Loading 0: 45%|████▌ | 348/771 [05:12<10:35, 1.50s/it]
Loading 0: 46%|████▌ | 351/771 [05:15<08:54, 1.27s/it]
Loading 0: 46%|████▌ | 352/771 [05:18<11:27, 1.64s/it]
Loading 0: 46%|████▌ | 353/771 [05:22<13:49, 1.98s/it]
Loading 0: 46%|████▌ | 356/771 [05:22<07:14, 1.05s/it]
Loading 0: 46%|████▋ | 357/771 [05:22<06:37, 1.04it/s]
Loading 0: 47%|████▋ | 359/771 [05:23<05:00, 1.37it/s]
Loading 0: 47%|████▋ | 361/771 [05:23<03:36, 1.90it/s]
Loading 0: 47%|████▋ | 363/771 [05:26<05:52, 1.16it/s]
Loading 0: 47%|████▋ | 364/771 [05:30<08:45, 1.29s/it]
Loading 0: 47%|████▋ | 365/771 [05:33<11:29, 1.70s/it]
Loading 0: 48%|████▊ | 368/771 [05:33<06:07, 1.10it/s]
Loading 0: 48%|████▊ | 369/771 [05:33<05:42, 1.17it/s]
Loading 0: 48%|████▊ | 371/771 [05:34<04:23, 1.52it/s]
Loading 0: 48%|████▊ | 373/771 [05:34<03:11, 2.08it/s]
Loading 0: 49%|████▊ | 375/771 [05:37<05:30, 1.20it/s]
Loading 0: 49%|████▉ | 376/771 [05:41<08:36, 1.31s/it]
Loading 0: 49%|████▉ | 377/771 [05:44<11:35, 1.76s/it]
Loading 0: 49%|████▉ | 380/771 [05:44<06:10, 1.05it/s]
Loading 0: 49%|████▉ | 381/771 [05:45<05:46, 1.13it/s]
Loading 0: 50%|████▉ | 383/771 [05:46<04:26, 1.46it/s]
Loading 0: 50%|████▉ | 385/771 [05:46<03:13, 1.99it/s]
Loading 0: 50%|█████ | 387/771 [05:49<05:22, 1.19it/s]
Loading 0: 50%|█████ | 388/771 [05:52<08:05, 1.27s/it]
Loading 0: 50%|█████ | 389/771 [05:55<10:37, 1.67s/it]
Loading 0: 51%|█████ | 392/771 [05:55<05:39, 1.12it/s]
Loading 0: 51%|█████ | 393/771 [05:56<05:17, 1.19it/s]
Loading 0: 51%|█████ | 395/771 [05:57<04:05, 1.53it/s]
Loading 0: 51%|█████▏ | 397/771 [05:57<02:57, 2.10it/s]
Loading 0: 52%|█████▏ | 398/771 [06:00<06:04, 1.02it/s]
Loading 0: 52%|█████▏ | 400/771 [06:00<04:02, 1.53it/s]
Loading 0: 52%|█████▏ | 401/771 [06:01<03:56, 1.57it/s]
Loading 0: 52%|█████▏ | 403/771 [06:01<03:08, 1.95it/s]
Loading 0: 53%|█████▎ | 405/771 [06:02<02:17, 2.66it/s]
Loading 0: 53%|█████▎ | 406/771 [06:02<02:01, 3.01it/s]
Loading 0: 53%|█████▎ | 407/771 [06:05<05:42, 1.06it/s]
Loading 0: 53%|█████▎ | 408/771 [06:08<08:55, 1.48s/it]
Loading 0: 53%|█████▎ | 411/771 [06:11<07:31, 1.25s/it]
Loading 0:
chaiml-mistral32-deeps-89261-v11-mkmlizer: quantized model in 521.462s
chaiml-mistral32-deeps-89261-v11-mkmlizer: Processed model ChaiML/mistral32_deepseek_cpos3_cpos21322_sfts11306_seq96_exp6 in 576.573s
chaiml-mistral32-deeps-89261-v11-mkmlizer: creating bucket guanaco-mkml-models
chaiml-mistral32-deeps-89261-v11-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-mistral32-deeps-89261-v11-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia
chaiml-mistral32-deeps-89261-v11-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia/config.json
chaiml-mistral32-deeps-89261-v11-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia/special_tokens_map.json
chaiml-mistral32-deeps-89261-v11-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia/tokenizer_config.json
chaiml-mistral32-deeps-89261-v11-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia/tokenizer.json
Job chaiml-mistral32-deeps-89261-v11-mkmlizer completed after 609.87s with status: succeeded
Stopping job with name chaiml-mistral32-deeps-89261-v11-mkmlizer
Pipeline stage MKMLizer completed in 612.52s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.84s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mistral32-deeps-89261-v11
Waiting for inference service chaiml-mistral32-deeps-89261-v11 to be ready
Inference service chaiml-nitral-dpos1-beta-5231-v7 ready after 303.3506917953491s
Pipeline stage MKMLDeployer completed in 307.32s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 4.129997968673706s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.932973861694336s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.566399097442627s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.5381314754486084s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 3.441862106323242s
5 requests
0 failed requests
5th percentile: 2.543784999847412
10th percentile: 2.5494385242462156
20th percentile: 2.5607455730438233
30th percentile: 2.6397140502929686
40th percentile: 2.7863439559936523
50th percentile: 2.932973861694336
60th percentile: 3.1365291595458986
70th percentile: 3.340084457397461
80th percentile: 3.5794892787933352
90th percentile: 3.8547436237335204
95th percentile: 3.992370796203613
99th percentile: 4.102472534179688
chaiml-mistral31-24b-s-69496-v24-mkmlizer: quantized model in 526.909s
mean time: 3.121872901916504
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Processed model ChaiML/mistral31-24b-simpoexp1-s1-new-sft-retryv2top20lex-2e in 607.278s
Pipeline stage StressChecker completed in 32.52s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: creating bucket guanaco-mkml-models
run pipeline stage %s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
chaiml-mistral31-24b-s-69496-v24-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.83s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/config.json
run pipeline stage %s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/special_tokens_map.json
Running pipeline stage TriggerMKMLProfilingPipeline
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/tokenizer_config.json
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.87s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/tokenizer.json
Shutdown handler de-registered
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/flywheel_model.1.safetensors
chaiml-nitral-dpos1-beta_5231_v7 status is now deployed due to DeploymentManager action
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-slerpv5-mistral-24540-v28-mkmlizer: quantized model in 525.841s
chaiml-slerpv5-mistral-24540-v28-mkmlizer: Processed model ChaiML/slerpv5_mistral31_simpo_dpo_dpoold_s2 in 599.466s
chaiml-slerpv5-mistral-24540-v28-mkmlizer: creating bucket guanaco-mkml-models
chaiml-slerpv5-mistral-24540-v28-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-slerpv5-mistral-24540-v28-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-slerpv5-mistral-24540-v28/nvidia
Job chaiml-mistral31-24b-s-69496-v24-mkmlizer completed after 644.18s with status: succeeded
chaiml-slerpv5-mistral-24540-v28-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-slerpv5-mistral-24540-v28/nvidia/config.json
Stopping job with name chaiml-mistral31-24b-s-69496-v24-mkmlizer
chaiml-slerpv5-mistral-24540-v28-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-slerpv5-mistral-24540-v28/nvidia/special_tokens_map.json
Pipeline stage MKMLizer completed in 647.32s
chaiml-slerpv5-mistral-24540-v28-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-slerpv5-mistral-24540-v28/nvidia/tokenizer_config.json
run pipeline stage %s
chaiml-slerpv5-mistral-24540-v28-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-slerpv5-mistral-24540-v28/nvidia/tokenizer.json
Running pipeline stage MKMLTemplater
chaiml-slerpv5-mistral-24540-v28-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-slerpv5-mistral-24540-v28/nvidia/flywheel_model.1.safetensors
Pipeline stage MKMLTemplater completed in 0.73s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mistral31-24b-s-69496-v24
Waiting for inference service chaiml-mistral31-24b-s-69496-v24 to be ready
Job chaiml-slerpv5-mistral-24540-v28-mkmlizer completed after 633.97s with status: succeeded
Stopping job with name chaiml-slerpv5-mistral-24540-v28-mkmlizer
Pipeline stage MKMLizer completed in 637.54s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.71s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-slerpv5-mistral-24540-v28
Waiting for inference service chaiml-slerpv5-mistral-24540-v28 to be ready
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: quantized model in 717.968s
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: Processed model ChaiML/cogito32b-dpos2-dpos1ft-b06-2e in 828.243s
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: creating bucket guanaco-mkml-models
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v8/nvidia
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v8/nvidia/config.json
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v8/nvidia/added_tokens.json
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v8/nvidia/special_tokens_map.json
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v8/nvidia/tokenizer_config.json
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: cp /dev/shm/model_cache/merges.txt s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v8/nvidia/merges.txt
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: cp /dev/shm/model_cache/vocab.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v8/nvidia/vocab.json
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-cogito32b-dpos2-39363-v8/nvidia/tokenizer.json
Job chaiml-cogito32b-dpos2-39363-v8-mkmlizer completed after 861.76s with status: succeeded
Stopping job with name chaiml-cogito32b-dpos2-39363-v8-mkmlizer
Pipeline stage MKMLizer completed in 864.26s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.70s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-cogito32b-dpos2-39363-v8
Waiting for inference service chaiml-cogito32b-dpos2-39363-v8 to be ready
Inference service chaiml-mistral32-deeps-89261-v11 ready after 303.46799969673157s
Pipeline stage MKMLDeployer completed in 307.06s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 4.210136413574219s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.7867519855499268s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.760551691055298s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.6716248989105225s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.8477282524108887s
5 requests
0 failed requests
5th percentile: 2.6894102573394774
10th percentile: 2.7071956157684327
20th percentile: 2.742766332626343
30th percentile: 2.7657917499542237
40th percentile: 2.776271867752075
50th percentile: 2.7867519855499268
60th percentile: 2.8111424922943113
70th percentile: 2.8355329990386964
80th percentile: 3.120209884643555
90th percentile: 3.665173149108887
95th percentile: 3.9376547813415526
99th percentile: 4.1556400871276855
mean time: 3.055358648300171
Pipeline stage StressChecker completed in 29.11s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.75s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.81s
Shutdown handler de-registered
chaiml-mistral32-deeps_89261_v11 status is now deployed due to DeploymentManager action
chaiml-mistral32-deeps_89261_v11 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2ee61815-10bc-42b7-815c-32418c76c4d8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:15:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b18d7ff9-49f2-4571-9527-b3460f8b70f0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:20:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'87e98ab5-219d-4bda-9b1b-45315d05a7ff\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:25:43 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b9631bc6-4799-41ae-be97-391897c48d4e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:30:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'11931fb2-488b-4471-b66b-3ceeb57bc0a3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:35:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a4280fca-2be4-443a-948e-4edd0300bf38\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:40:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'11b95f8e-4a79-42e8-9528-d8e176a156e1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:45:48 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2de57472-484b-454d-afb0-c1862588edd0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:50:46 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ae4916d1-17dd-4979-a9d1-1ca473153396\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:55:48 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'aadb9ef9-c29e-47f4-a2a9-9f4a228bd512\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:00:47 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'96a88fc3-d6ad-45dc-aa6a-caf08cf189f2\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:05:47 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5944f5fe-5400-411b-8307-060988031755\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:10:48 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'90d51a1d-7d82-48cc-80d6-c159572f1c34\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:15:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'198b41fe-facd-4926-be30-0a71c46834e5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:20:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f0f4ee99-a13b-48e2-ab63-ee03a93454f8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:25:54 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8a728665-e12f-4ba1-abb9-edbc661b9141\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:30:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2437a4f7-4236-48c9-8830-a2fb822bcf5b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:35:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4277a295-06de-48ab-bde6-b07874a60053\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:40:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3630ffe3-b4de-4e8d-88d9-e6df186d336d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:45:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d4734984-0f0b-436a-95cc-ba3ad735c435\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:50:54 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'96854642-eafe-46e4-988a-90be28d4b79c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:55:55 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ae9c525f-fde7-407e-8ffe-2e303c5d579b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:00:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'172dff41-116a-454a-8ac6-5077993cac6e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:05:56 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5db17976-b9df-404b-b73e-2edfe73b0446\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:10:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2e12c1da-f214-47bc-b241-1ddae6b7471d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:16:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b3776ffd-3ce3-4a94-a815-df1e37cbccc2\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:20:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ee282930-6fbc-4cd1-8762-41b1285f9307\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:26:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'de4e461f-e9d3-4d55-b8a0-35597736c346\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:30:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8dd965a0-36b5-4ed3-b913-1a2e0ae9144f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:36:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'90d30b0e-91f8-4955-b750-28b684d5df1d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:40:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3918a564-bc70-4feb-8ddb-1184cd1b76f7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:45:56 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'17c3a925-72e1-405a-a332-98b21b08cebd\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:50:55 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c42bceb7-ef1e-47d7-b1c5-5cef6be02e22\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:55:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'966030a2-9196-478a-9d25-a827e1b164e4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:00:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3c9b932b-6eaa-454b-a243-23c359aedec4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:05:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'125a1c8f-756b-4406-b44c-ebf69555849a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:10:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f119db31-4146-417b-9bd9-8416556ebd50\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:16:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1abbfb4f-7958-4c8b-9cfb-4a96340ab0bc\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:21:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d298cec6-4a0f-4353-ada5-2bc4158a58c4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:26:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'71283948-c393-4103-9fba-9c6ff4f565fa\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:31:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'440194fe-1306-4c33-b9b2-745b57c1665d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:36:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'10efd598-8a90-4545-92d9-e3dd4591b4c0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:41:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7b361501-6bc0-41e8-a206-3a580e37632a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:46:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a757febb-d78d-4e9f-934d-c94d42a43d1d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:51:09 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6814ea59-4875-40d7-9136-ea8a470363f4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:56:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fba962a9-3588-4251-9f10-f88c97fa7eac\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:01:05 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fcdce7e9-6bcd-4812-b7d3-ae26bf260fd7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:06:07 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0cedd20c-7636-4e71-8774-aa920e4ca2e0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:11:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'84acab89-9bd2-474a-9f19-38356124845e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:16:07 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2e0258bd-6dd7-4af7-9e7e-a6f36aab1871\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:21:10 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5a8e1d17-4388-433f-8232-082f18ccbb4f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:26:12 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2095fc47-1d30-40f5-ad6c-bc06ddeee627\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:31:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c68bc9ee-7897-4d3c-8d87-f0f9555e21c3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:36:12 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e14c6c36-9ca5-42c1-87ec-5bfdd54e1818\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:41:09 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2a256245-0e32-481f-8bd6-2d0e1698bb98\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:46:07 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d606ce35-1680-4634-a816-db171c653f33\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:51:14 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a2686e01-a7b1-4328-b749-9441aac45b18\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:56:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7f1f63fe-1dd5-4416-ba41-bfd3c3154ed8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:01:18 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b990920c-8620-43d1-b6b9-180dcf8b1089\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:06:14 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'73794989-a48d-4680-9df2-517371be3aca\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:11:15 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'70e6b6e6-f6d3-4045-8a47-80e48cd3cba6\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:16:19 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'03e02182-75f4-422d-91c6-ec85b356d0d9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:21:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d7c80316-1e04-46ac-a7d3-fdaa82664b28\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:26:12 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f243e9c7-e84e-474b-a2a9-0db752b50229\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:31:16 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9ff1abba-9d78-4f3b-aafe-989f372ab267\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:36:18 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6213bae3-7b50-438f-a30a-7bd32220f0ca\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:41:18 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6ddc1df8-7081-4657-96ae-9f948f86c70f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:46:17 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'72c85313-7319-423c-989e-5473cc9726bf\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:51:17 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'83f2cd7f-5d2f-4bd1-ad51-3592984677b1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:56:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3e7679b2-3d57-485c-ad2e-4bc5fce3dbcf\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:01:26 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ee98b0ec-f001-4480-bf56-4762e268bc6e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:06:28 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'03bb38da-714d-4b2a-ad60-38dffac9f799\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:11:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f26e905b-4636-4a0d-a230-1b9b09fd33e6\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:16:21 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'af3d6996-3f8b-48a4-9b19-3901505a5771\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:21:24 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bccd3f41-2fb3-4900-bc98-1c9da004dcf1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:26:21 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'633558fa-812c-48c6-b920-00b6ab25ce84\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:31:28 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c5b809cb-0f26-4257-bb58-c985230fcefe\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:36:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8bd11b86-d658-4079-8271-9cbe0d82060c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:41:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8c538e7d-9bef-4604-b40b-896f9e4c1d08\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:46:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c4d31deb-2e4e-4028-970c-c2ce2f145693\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:51:28 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'51ed6e40-62a6-42bf-889b-c034c455930a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:56:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dd11559a-38dd-4fa3-8e29-bf60ed480d49\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:01:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'54c9e607-1bab-479b-9287-ed2b5c0f641d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:06:26 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
HTTP Request: %s %s "%s %d %s"
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'97db8df1-776e-47d0-9357-05dc2fb2b4be\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:11:28 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'15081e64-6cba-4532-bd3e-9227fe11e98c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:16:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e43f08e2-2a6f-421e-8cc5-58124f1d8300\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:21:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6e675bb0-fcbc-4415-814b-7fda5b1b2e93\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:26:28 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8b0a1b9c-c9ff-41ce-83a8-218b45df3e67\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:31:50 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f96bdf2f-375b-4b11-b7c4-92b6dc8ddf5d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:37:06 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'05201988-086b-4486-a851-7d6918e62acd\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:41:52 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'07269d80-9020-455f-82e4-0245cdcb256d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:46:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'80b888f3-8246-4227-ae70-efebc004030c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:51:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a448ab8b-69fa-4851-8f31-43e6e6599d3f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:56:34 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6c9c6d88-8a18-4eca-85b8-7505ea7ee801\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:01:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2b747988-743f-4883-9a23-3171a63b3365\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:06:48 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'12b1b814-6740-4ea9-b41c-b36d670a541f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:11:50 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f7d5e0e1-daf5-447c-97cb-761853f34d38\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:16:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'35341617-663f-4a8f-9f58-1bd424a14654\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:21:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e376b7a4-05c2-4b7d-8eb3-684a375dffb1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:26:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'27e98551-2cac-4f56-8c42-328a4f281a19\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:31:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cb60319b-2e60-4106-af25-be4752b88b0e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:36:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'22cde71b-1746-47e5-8b27-3a7d90c15832\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:42:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6ad124ed-699e-4564-88e3-34bad9bba716\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:51:47 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'05c9623e-d16f-4021-a595-3e4f9ed169db\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:57:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'49c0ee56-72e1-4930-b1ed-16c6f9305c93\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:02:15 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
HTTP Request: %s %s "%s %d %s"
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4609f226-f54e-4d51-ad27-2d113c28c197\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:07:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
HTTP Request: %s %s "%s %d %s"
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'19e605a5-d923-4b42-bd95-e6bf2ebf9373\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:16:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'95db57b6-151f-4513-ba91-7370d647dace\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:22:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c54f6001-64bd-4a3e-bb81-427d872f5cbb\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:27:38 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dbb915d1-3c9a-4524-8a57-6851b5f6ea47\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:36:53 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1407eb4f-76a6-4269-b8da-8fa54d5558b4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:42:15 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b9e7b781-48fb-4c91-82cb-e412d609fc59\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:47:50 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f8c47b26-a094-43c2-92c6-edc50a8b5a23\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:56:47 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'34db7340-2428-44b8-837f-31d450f72ceb\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:02:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
HTTP Request: %s %s "%s %d %s"
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b6d2b1b0-be21-4120-a1f8-767693b0cae1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:08:06 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'223e7d6b-4b49-4980-bde2-5bd06319b6f6\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:16:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fc7d9ca1-0173-4796-80b0-5cad545e5979\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:22:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bb1c0c8e-05f4-4527-aa38-d41743b32d29\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:32:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'73026f53-a3c5-4156-8bd5-95aba04ab131\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:37:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0f38ecff-89f3-4fc3-a13a-8445eb44e3ee\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:46:52 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ef19f911-435b-46e0-a2c7-5b3905e3e057\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:52:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'28403e70-ace6-48dd-9c37-b2dffa00f434\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:02:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a32774e2-3690-4d7f-8017-9bd834e00c40\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:07:40 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9b7cd39a-136a-4b81-89a8-64ef2d4e143f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:17:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7b0fff00-e0f1-4887-9ed0-33cb9fb26efb\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:22:48 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ffd3357b-c26c-47c0-b254-4504a705d050\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:32:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'738760cf-72b2-4c23-96f1-08893cd9bc76\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:37:49 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cbea15c8-175f-4fb9-86f3-91dad3cde61d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:47:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9c07afbf-4650-40ba-afc1-2781376c32ff\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:52:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c382c5e6-07e1-46e4-979f-3f9110d7aaad\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:02:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'79f20f67-3223-4f43-a2cf-b671e38b6183\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:07:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d10397f9-dd26-44d7-92e8-c1cafe983eb6\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:17:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'44cc1b08-7090-4d4d-855b-4c5b800ebbc8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:23:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1c3244c1-065f-4eb1-a55d-5ae7c6f8ebe6\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:32:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'330110d0-0420-4334-811e-7a74811889f9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:38:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Failed to get response for submission blend_rurun_2025-08-28: ('http://guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com/models/chaiml-rirv938-0sw-96p-_18713_v1/predict', '{"detail":"list index out of range"}')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9690325f-a164-4aa6-8ead-4e55f990f951\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:47:10 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'453c8d43-4307-4c6f-b974-386c94f73e5d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:52:54 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'54985052-cc7e-4629-ab20-e539d3da7e99\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:02:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f1cd98c8-5c30-4931-ac7f-0e0b37b4963c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:07:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c36320e2-8ab5-4c39-ae19-e8dbff19a921\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:17:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2afc044c-60ac-42ee-994a-4507dbb97f11\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:23:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9c8c7802-d459-4ef6-81dc-e078d22ad26e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:32:12 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'feea6de6-d8ad-4f9c-8088-21021673ab41\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:38:16 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c2110494-fb32-4c51-8456-f322ccc3127d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:47:26 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'58fe62c2-aabc-4818-90f7-325271d1f6e5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:53:16 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'793f02ae-c59e-4148-a1de-75bda3f18643\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:58:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'aae93cee-432b-4b28-926f-42e9afbce4fe\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:07:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'06de39bf-15d3-460c-b356-e85c15617539\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:13:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2b27c58e-cf0d-47b5-aa4a-14076e4714bc\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:18:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b7b90adc-9ca8-4c9f-b8a1-7a7fa6426994\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:27:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'889dc7ec-0fb6-48f5-b050-21b412352732\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:33:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'41b3f407-638b-4ec1-99ce-33ec8884029b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:38:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9f0e2d24-27d8-43b6-8b9c-9dc3376ff629\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:47:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'92664680-1b8d-4139-9f66-80c080c905cd\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:53:29 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'002e4dfe-3062-4e1b-ae4d-e9d2a3f9e94d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 15:22:26 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f8d9fc3b-ad73-4ad5-b657-fe836d38545e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 15:28:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9fdce32c-9a4a-4a7f-996c-6fc962fe3ac5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 16:03:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
HTTP Request: %s %s "%s %d %s"
admin requested tearing down of chaiml-mistral32-deeps_89261_v11
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cb115667-e009-475f-a897-ae76d954e45f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 16:18:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered