Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-nitral-dpos1-beta-5231-v7-mkmlizer
Waiting for job on chaiml-nitral-dpos1-beta-5231-v7-mkmlizer to finish
HTTP Request: %s %s "%s %d %s"
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-cogito32b-dpos2-39363-v8-mkmlizer
HTTP Request: %s %s "%s %d %s"
Waiting for job on chaiml-cogito32b-dpos2-39363-v8-mkmlizer to finish
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ Version: 0.29.15 ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ https://mk1.ai ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ belonging to: ║
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ Chai Research Corp. ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
HTTP Request: %s %s "%s %d %s"
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Shutdown handler not registered because Python interpreter is not running in the main thread
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
run pipeline %s
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
run pipeline stage %s
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Running pipeline stage MKMLizer
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Starting job with name chaiml-mistral32-deeps-89261-v11-mkmlizer
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Waiting for job on chaiml-mistral32-deeps-89261-v11-mkmlizer to finish
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
HTTP Request: %s %s "%s %d %s"
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Version: 0.29.15 ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ https://mk1.ai ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ The license key for the current software has been verified as ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ belonging to: ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Chai Research Corp. ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ║ ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Version: 0.29.15 ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ https://mk1.ai ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ The license key for the current software has been verified as ║
1 validation error for RewardSubmission
platform
Input should be 'reward' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ belonging to: ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Chai Research Corp. ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ║ ║
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral32-deeps-89261-v11-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Shutdown handler not registered because Python interpreter is not running in the main thread
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
run pipeline %s
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
run pipeline stage %s
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Running pipeline stage MKMLizer
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Starting job with name chaiml-mistral31-24b-s-69496-v24-mkmlizer
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
Waiting for job on chaiml-mistral31-24b-s-69496-v24-mkmlizer to finish
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for GuardSubmission
platform
Input should be 'guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Downloaded to shared memory in 44.064s
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Checking if ChaiML/nitral-dpos1-beta06-exp1-10kpref-1800-lowlr-2e-2-1 already exists in ChaiML
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmpooxfu2hy, device:0
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Version: 0.29.15 ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ https://mk1.ai ║
1 validation error for NemoGuardSubmission
platform
Input should be 'nemo_guard' [type=literal_error, input_value='coreweave/mkml', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/literal_error
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ belonging to: ║
Shutdown handler not registered because Python interpreter is not running in the main thread
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
run pipeline %s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Chai Research Corp. ║
run pipeline stage %s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
Running pipeline stage MKMLizer
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
Starting job with name chaiml-slerpv5-mistral-24540-v28-mkmlizer
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ║ ║
chaiml-mistral31-24b-s-69496-v24-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Waiting for job on chaiml-slerpv5-mistral-24540-v28-mkmlizer to finish
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Version: 0.29.15 ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ https://mk1.ai ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ belonging to: ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Chai Research Corp. ║
chaiml-mistral32-deeps-89261-v11-mkmlizer: Downloaded to shared memory in 55.110s
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-mistral32-deeps-89261-v11-mkmlizer: Checking if ChaiML/mistral32_deepseek_cpos3_cpos21322_sfts11306_seq96_exp6 already exists in ChaiML
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-mistral32-deeps-89261-v11-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmpi8v_yyxi, device:0
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ║ ║
chaiml-mistral32-deeps-89261-v11-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-slerpv5-mistral-24540-v28-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: Downloaded to shared memory in 110.274s
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: Checking if ChaiML/cogito32b-dpos2-dpos1ft-b06-2e already exists in ChaiML
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmpisle3d3p, device:0
chaiml-cogito32b-dpos2-39363-v8-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Downloaded to shared memory in 80.368s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Checking if ChaiML/mistral31-24b-simpoexp1-s1-new-sft-retryv2top20lex-2e already exists in ChaiML
chaiml-mistral31-24b-s-69496-v24-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmpfv82i4tl, device:0
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-slerpv5-mistral-24540-v28-mkmlizer: Downloaded to shared memory in 73.625s
chaiml-slerpv5-mistral-24540-v28-mkmlizer: Checking if ChaiML/slerpv5_mistral31_simpo_dpo_dpoold_s2 already exists in ChaiML
chaiml-slerpv5-mistral-24540-v28-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q3z, folder:/tmp/tmp3ybje2zs, device:0
chaiml-slerpv5-mistral-24540-v28-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: quantized model in 259.199s
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Processed model ChaiML/nitral-dpos1-beta06-exp1-10kpref-1800-lowlr-2e-2-1 in 303.264s
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nitral-dpos1-beta-5231-v7/nvidia
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nitral-dpos1-beta-5231-v7/nvidia/config.json
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nitral-dpos1-beta-5231-v7/nvidia/special_tokens_map.json
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nitral-dpos1-beta-5231-v7/nvidia/flywheel_model.0.safetensors
chaiml-nitral-dpos1-beta-5231-v7-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 3/363 [00:01<03:22, 1.78it/s]
Loading 0: 1%| | 4/363 [00:03<05:34, 1.07it/s]
Loading 0: 1%|▏ | 5/363 [00:05<07:13, 1.21s/it]
Loading 0: 2%|▏ | 7/363 [00:05<03:56, 1.51it/s]
Loading 0: 2%|▏ | 8/363 [00:05<03:38, 1.62it/s]
Loading 0: 2%|▏ | 9/363 [00:06<03:24, 1.73it/s]
Loading 0: 3%|▎ | 10/363 [00:06<02:45, 2.14it/s]
Loading 0: 3%|▎ | 12/363 [00:08<03:40, 1.59it/s]
Loading 0: 4%|▎ | 13/363 [00:09<05:07, 1.14it/s]
Loading 0: 4%|▍ | 14/363 [00:11<06:28, 1.11s/it]
Loading 0: 4%|▍ | 16/363 [00:11<03:51, 1.50it/s]
Loading 0: 5%|▍ | 17/363 [00:12<03:34, 1.61it/s]
Loading 0: 5%|▍ | 18/363 [00:12<03:21, 1.72it/s]
Loading 0: 5%|▌ | 19/363 [00:12<02:45, 2.08it/s]
Loading 0: 6%|▌ | 21/363 [00:14<03:34, 1.60it/s]
Loading 0: 6%|▌ | 22/363 [00:16<04:58, 1.14it/s]
Loading 0: 6%|▋ | 23/363 [00:17<06:16, 1.11s/it]
Loading 0: 7%|▋ | 25/363 [00:18<03:46, 1.49it/s]
Loading 0: 7%|▋ | 26/363 [00:18<03:31, 1.60it/s]
Loading 0: 7%|▋ | 27/363 [00:19<03:19, 1.68it/s]
Loading 0: 8%|▊ | 28/363 [00:19<02:43, 2.05it/s]
Loading 0: 8%|▊ | 30/363 [00:20<03:31, 1.58it/s]
Loading 0: 9%|▊ | 31/363 [00:22<04:50, 1.14it/s]
Loading 0: 9%|▉ | 32/363 [00:24<06:05, 1.10s/it]
Loading 0: 9%|▉ | 34/363 [00:24<03:39, 1.50it/s]
Loading 0: 10%|▉ | 35/363 [00:24<03:24, 1.60it/s]
Loading 0: 10%|▉ | 36/363 [00:25<03:12, 1.70it/s]
Loading 0: 10%|█ | 37/363 [00:25<02:37, 2.07it/s]
Loading 0: 11%|█ | 39/363 [00:27<03:24, 1.58it/s]
Loading 0: 11%|█ | 40/363 [00:28<04:42, 1.14it/s]
Loading 0: 11%|█▏ | 41/363 [00:30<05:50, 1.09s/it]
Loading 0: 12%|█▏ | 43/363 [00:30<03:30, 1.52it/s]
Loading 0: 12%|█▏ | 44/363 [00:31<03:17, 1.62it/s]
Loading 0: 12%|█▏ | 45/363 [00:31<03:06, 1.70it/s]
Loading 0: 13%|█▎ | 46/363 [00:31<02:32, 2.07it/s]
Loading 0: 13%|█▎ | 48/363 [00:33<03:17, 1.59it/s]
Loading 0: 13%|█▎ | 49/363 [00:35<04:32, 1.15it/s]
Loading 0: 14%|█▍ | 50/363 [00:37<05:41, 1.09s/it]
Loading 0: 14%|█▍ | 52/363 [00:37<03:24, 1.52it/s]
Loading 0: 15%|█▍ | 53/363 [00:37<03:11, 1.62it/s]
Loading 0: 15%|█▍ | 54/363 [00:38<03:00, 1.72it/s]
Loading 0: 15%|█▌ | 55/363 [00:38<02:26, 2.11it/s]
Loading 0: 15%|█▌ | 56/363 [00:39<04:04, 1.26it/s]
Loading 0: 16%|█▌ | 57/363 [00:40<03:05, 1.65it/s]
Loading 0: 16%|█▌ | 58/363 [00:40<02:51, 1.77it/s]
Loading 0: 16%|█▋ | 59/363 [00:40<02:44, 1.85it/s]
Loading 0: 17%|█▋ | 60/363 [00:41<02:13, 2.27it/s]
Loading 0: 17%|█▋ | 61/363 [00:41<01:52, 2.70it/s]
Loading 0: 17%|█▋ | 62/363 [00:43<03:46, 1.33it/s]
Loading 0: 17%|█▋ | 63/363 [00:44<05:08, 1.03s/it]
Loading 0: 18%|█▊ | 64/363 [00:46<06:12, 1.24s/it]
Loading 0: 18%|█▊ | 66/363 [00:46<03:26, 1.44it/s]
Loading 0: 18%|█▊ | 67/363 [00:47<03:09, 1.57it/s]
Loading 0: 19%|█▊ | 68/363 [00:47<02:56, 1.67it/s]
Loading 0: 19%|█▉ | 69/363 [00:47<02:23, 2.06it/s]
Loading 0: 20%|█▉ | 71/363 [00:49<03:08, 1.55it/s]
Loading 0: 20%|█▉ | 72/363 [00:51<04:20, 1.12it/s]
Loading 0: 20%|██ | 73/363 [00:52<05:24, 1.12s/it]
Loading 0: 21%|██ | 75/363 [00:52<03:13, 1.49it/s]
Loading 0: 21%|██ | 76/363 [00:53<02:59, 1.60it/s]
Loading 0: 21%|██ | 77/363 [00:53<02:47, 1.70it/s]
Loading 0: 21%|██▏ | 78/363 [00:54<02:15, 2.10it/s]
Loading 0: 22%|██▏ | 80/363 [00:55<02:59, 1.58it/s]
Loading 0: 22%|██▏ | 81/363 [00:57<04:11, 1.12it/s]
Loading 0: 23%|██▎ | 82/363 [00:59<05:15, 1.12s/it]
Loading 0: 23%|██▎ | 84/363 [00:59<03:08, 1.48it/s]
Loading 0: 23%|██▎ | 85/363 [00:59<02:56, 1.57it/s]
Loading 0: 24%|██▎ | 86/363 [01:00<02:46, 1.66it/s]
Loading 0: 24%|██▍ | 87/363 [01:00<02:17, 2.01it/s]
Loading 0: 25%|██▍ | 89/363 [01:02<02:55, 1.56it/s]
Loading 0: 25%|██▍ | 90/363 [01:04<04:07, 1.10it/s]
Loading 0: 25%|██▌ | 91/363 [01:05<05:11, 1.15s/it]
Loading 0: 26%|██▌ | 93/363 [01:06<03:07, 1.44it/s]
Loading 0: 26%|██▌ | 94/363 [01:06<02:55, 1.53it/s]
Loading 0: 26%|██▌ | 95/363 [01:07<02:44, 1.63it/s]
Loading 0: 26%|██▋ | 96/363 [01:07<02:14, 1.98it/s]
Loading 0: 27%|██▋ | 98/363 [01:08<02:53, 1.53it/s]
Loading 0: 27%|██▋ | 99/363 [01:10<03:57, 1.11it/s]
Loading 0: 28%|██▊ | 100/363 [01:12<04:53, 1.12s/it]
Loading 0: 28%|██▊ | 102/363 [01:12<02:55, 1.49it/s]
Loading 0: 28%|██▊ | 103/363 [01:13<02:43, 1.59it/s]
Loading 0: 29%|██▊ | 104/363 [01:13<02:32, 1.69it/s]
Loading 0: 29%|██▉ | 105/363 [01:13<02:03, 2.09it/s]
Loading 0: 29%|██▉ | 106/363 [01:15<03:24, 1.25it/s]
Loading 0: 29%|██▉ | 107/363 [01:15<02:34, 1.65it/s]
Loading 0: 30%|██▉ | 108/363 [01:15<02:24, 1.76it/s]
Loading 0: 30%|███ | 109/363 [01:16<02:17, 1.85it/s]
Loading 0: 30%|███ | 110/363 [01:16<01:49, 2.31it/s]
Loading 0: 31%|███ | 112/363 [01:18<02:30, 1.67it/s]
Loading 0: 31%|███ | 113/363 [01:19<03:35, 1.16it/s]
Loading 0: 32%|███▏ | 116/363 [01:21<02:51, 1.44it/s]
Loading 0: 32%|███▏ | 117/363 [01:23<03:35, 1.14it/s]
Loading 0: 33%|███▎ | 118/363 [01:24<04:19, 1.06s/it]
Loading 0: 33%|███▎ | 120/363 [01:24<02:43, 1.49it/s]
Loading 0: 33%|███▎ | 121/363 [01:25<02:32, 1.59it/s]
Loading 0: 34%|███▎ | 122/363 [01:25<02:22, 1.69it/s]
Loading 0: 34%|███▍ | 123/363 [01:25<01:56, 2.05it/s]
Loading 0: 34%|███▍ | 125/363 [01:27<02:28, 1.61it/s]
Loading 0: 35%|███▍ | 126/363 [01:29<03:22, 1.17it/s]
Loading 0: 35%|███▍ | 127/363 [01:30<04:09, 1.06s/it]
Loading 0: 36%|███▌ | 129/363 [01:31<02:30, 1.56it/s]
Loading 0: 36%|███▌ | 130/363 [01:31<02:19, 1.67it/s]
Loading 0: 36%|███▌ | 131/363 [01:31<02:11, 1.77it/s]
Loading 0: 36%|███▋ | 132/363 [01:32<01:47, 2.16it/s]
Loading 0: 37%|███▋ | 134/363 [01:33<02:19, 1.64it/s]
Loading 0: 37%|███▋ | 135/363 [01:35<03:13, 1.18it/s]
Loading 0: 37%|███▋ | 136/363 [01:37<04:00, 1.06s/it]
Loading 0: 38%|███▊ | 138/363 [01:37<02:24, 1.56it/s]
Loading 0: 38%|███▊ | 139/363 [01:37<02:14, 1.66it/s]
Loading 0: 39%|███▊ | 140/363 [01:38<02:06, 1.76it/s]
Loading 0: 39%|███▉ | 141/363 [01:38<01:42, 2.16it/s]
Loading 0: 39%|███▉ | 142/363 [01:38<01:24, 2.62it/s]
Loading 0: 39%|███▉ | 143/363 [01:40<02:40, 1.37it/s]
Loading 0: 40%|███▉ | 144/363 [01:41<03:41, 1.01s/it]
Loading 0: 40%|████ | 147/363 [01:43<02:41, 1.34it/s]
Loading 0: 41%|████ | 148/363 [01:45<03:20, 1.07it/s]
Loading 0: 41%|████ | 149/363 [01:46<03:58, 1.11s/it]
Loading 0: 42%|████▏ | 151/363 [01:46<02:27, 1.44it/s]
Loading 0: 42%|████▏ | 152/363 [01:47<02:15, 1.56it/s]
Loading 0: 42%|████▏ | 153/363 [01:47<02:06, 1.66it/s]
Loading 0: 42%|████▏ | 154/363 [01:47<01:43, 2.02it/s]
Loading 0: 43%|████▎ | 156/363 [01:49<02:10, 1.59it/s]
Loading 0: 43%|████▎ | 157/363 [01:51<02:57, 1.16it/s]
Loading 0: 44%|████▎ | 158/363 [01:52<03:39, 1.07s/it]
Loading 0: 44%|████▍ | 160/363 [01:53<02:11, 1.54it/s]
Loading 0: 44%|████▍ | 161/363 [01:53<02:01, 1.66it/s]
Loading 0: 45%|████▍ | 162/363 [01:53<01:54, 1.75it/s]
Loading 0: 45%|████▍ | 163/363 [01:54<01:33, 2.15it/s]
Loading 0: 45%|████▌ | 165/363 [01:55<02:00, 1.64it/s]
Loading 0: 46%|████▌ | 166/363 [01:57<02:47, 1.18it/s]
Loading 0: 46%|████▌ | 167/363 [01:59<03:29, 1.07s/it]
Loading 0: 47%|████▋ | 169/363 [01:59<02:05, 1.55it/s]
Loading 0: 47%|████▋ | 170/363 [01:59<01:55, 1.67it/s]
Loading 0: 47%|████▋ | 171/363 [02:00<01:48, 1.76it/s]
Loading 0: 47%|████▋ | 172/363 [02:00<01:28, 2.16it/s]
Loading 0: 48%|████▊ | 174/363 [02:01<01:55, 1.64it/s]
Loading 0: 48%|████▊ | 175/363 [02:03<02:38, 1.18it/s]
Loading 0: 48%|████▊ | 176/363 [02:05<03:17, 1.06s/it]
Loading 0: 49%|████▉ | 178/363 [02:05<01:57, 1.57it/s]
Loading 0: 49%|████▉ | 179/363 [02:05<01:49, 1.68it/s]
Loading 0: 50%|████▉ | 180/363 [02:06<01:43, 1.77it/s]
Loading 0: 50%|████▉ | 181/363 [02:06<01:24, 2.15it/s]
Loading 0: 50%|█████ | 183/363 [02:07<01:49, 1.65it/s]
Loading 0: 51%|█████ | 184/363 [02:09<02:31, 1.18it/s]
Loading 0: 51%|█████ | 185/363 [02:11<03:06, 1.05s/it]
Loading 0: 52%|█████▏ | 187/363 [02:11<01:51, 1.58it/s]
Loading 0: 52%|█████▏ | 188/363 [02:11<01:43, 1.69it/s]
Loading 0: 52%|█████▏ | 189/363 [02:12<01:37, 1.78it/s]
Loading 0: 52%|█████▏ | 190/363 [02:12<01:20, 2.15it/s]
Loading 0: 53%|█████▎ | 192/363 [02:14<01:43, 1.65it/s]
Loading 0: 53%|█████▎ | 193/363 [02:15<02:23, 1.19it/s]
Loading 0: 53%|█████▎ | 194/363 [02:17<02:58, 1.06s/it]
Loading 0: 54%|█████▍ | 196/363 [02:17<01:46, 1.57it/s]
Loading 0: 54%|█████▍ | 197/363 [02:17<01:38, 1.69it/s]
Loading 0: 55%|█████▍ | 198/363 [02:18<01:32, 1.78it/s]
Loading 0: 55%|█████▍ | 199/363 [02:18<01:15, 2.18it/s]
Loading 0: 55%|█████▌ | 201/363 [02:20<01:38, 1.65it/s]
Loading 0: 56%|█████▌ | 202/363 [02:21<02:15, 1.19it/s]
Loading 0: 56%|█████▌ | 203/363 [02:23<02:48, 1.05s/it]
Loading 0: 56%|█████▋ | 205/363 [02:23<01:40, 1.57it/s]
Loading 0: 57%|█████▋ | 206/363 [02:24<01:33, 1.68it/s]
Loading 0: 57%|█████▋ | 207/363 [02:24<01:27, 1.79it/s]
Loading 0: 57%|█████▋ | 208/363 [02:24<01:10, 2.19it/s]
Loading 0: 58%|█████▊ | 210/363 [02:26<01:32, 1.65it/s]
Loading 0: 58%|█████▊ | 211/363 [02:27<02:08, 1.18it/s]
Loading 0: 58%|█████▊ | 212/363 [02:29<02:38, 1.05s/it]
Loading 0: 59%|█████▉ | 214/363 [02:29<01:34, 1.57it/s]
Loading 0: 59%|█████▉ | 215/363 [02:30<01:27, 1.69it/s]
Loading 0: 60%|█████▉ | 216/363 [02:30<01:22, 1.78it/s]
Loading 0: 60%|█████▉ | 217/363 [02:30<01:07, 2.16it/s]
Loading 0: 60%|██████ | 218/363 [02:32<01:53, 1.28it/s]
Loading 0: 60%|██████ | 219/363 [02:32<01:25, 1.69it/s]
Loading 0: 61%|██████ | 220/363 [02:32<01:18, 1.82it/s]
Loading 0: 61%|██████ | 221/363 [02:33<01:14, 1.90it/s]
Loading 0: 61%|██████ | 222/363 [02:33<00:59, 2.37it/s]
Loading 0: 61%|██████▏ | 223/363 [02:33<00:49, 2.85it/s]
Loading 0: 62%|██████▏ | 224/363 [02:35<01:40, 1.38it/s]
Loading 0: 62%|██████▏ | 225/363 [02:37<02:19, 1.01s/it]
Loading 0: 63%|██████▎ | 228/363 [02:38<01:41, 1.33it/s]
Loading 0: 63%|██████▎ | 229/363 [02:40<02:04, 1.07it/s]
Loading 0: 63%|██████▎ | 230/363 [02:42<02:27, 1.11s/it]
Loading 0: 64%|██████▍ | 232/363 [02:42<01:30, 1.45it/s]
Loading 0: 64%|██████▍ | 233/363 [02:42<01:22, 1.57it/s]
Loading 0: 64%|██████▍ | 234/363 [02:43<01:16, 1.68it/s]
Loading 0: 65%|██████▍ | 235/363 [02:43<01:02, 2.06it/s]
Loading 0: 65%|██████▌ | 237/363 [02:44<01:18, 1.61it/s]
Loading 0: 66%|██████▌ | 238/363 [02:46<01:46, 1.17it/s]
Loading 0: 66%|██████▌ | 239/363 [02:48<02:11, 1.06s/it]
Loading 0: 66%|██████▋ | 241/363 [02:48<01:18, 1.55it/s]
Loading 0: 67%|██████▋ | 242/363 [02:48<01:12, 1.67it/s]
Loading 0: 67%|██████▋ | 243/363 [02:49<01:07, 1.78it/s]
Loading 0: 67%|██████▋ | 244/363 [02:49<00:55, 2.16it/s]
Loading 0: 68%|██████▊ | 246/363 [02:50<01:11, 1.64it/s]
Loading 0: 68%|██████▊ | 247/363 [02:52<01:38, 1.18it/s]
Loading 0: 68%|██████▊ | 248/363 [02:54<02:01, 1.05s/it]
Loading 0: 69%|██████▉ | 250/363 [02:54<01:12, 1.57it/s]
Loading 0: 69%|██████▉ | 251/363 [02:54<01:06, 1.68it/s]
Loading 0: 69%|██████▉ | 252/363 [02:55<01:02, 1.78it/s]
Loading 0: 70%|██████▉ | 253/363 [02:55<00:50, 2.18it/s]
Loading 0: 70%|███████ | 255/363 [02:57<01:05, 1.66it/s]
Loading 0: 71%|███████ | 256/363 [02:58<01:30, 1.19it/s]
Loading 0: 71%|███████ | 257/363 [03:00<01:51, 1.05s/it]
Loading 0: 71%|███████▏ | 259/363 [03:00<01:06, 1.57it/s]
Loading 0: 72%|███████▏ | 260/363 [03:00<01:01, 1.67it/s]
Loading 0: 72%|███████▏ | 261/363 [03:01<00:57, 1.77it/s]
Loading 0: 72%|███████▏ | 262/363 [03:01<00:46, 2.17it/s]
Loading 0: 73%|███████▎ | 264/363 [03:03<01:00, 1.65it/s]
Loading 0: 73%|███████▎ | 265/363 [03:04<01:22, 1.18it/s]
Loading 0: 73%|███████▎ | 266/363 [03:06<01:42, 1.05s/it]
Loading 0: 74%|███████▍ | 268/363 [03:06<01:00, 1.57it/s]
Loading 0: 74%|███████▍ | 269/363 [03:07<00:55, 1.69it/s]
Loading 0: 74%|███████▍ | 270/363 [03:07<00:52, 1.78it/s]
Loading 0: 75%|███████▍ | 271/363 [03:07<00:42, 2.18it/s]
Loading 0: 75%|███████▌ | 273/363 [03:09<00:54, 1.65it/s]
Loading 0: 75%|███████▌ | 274/363 [03:10<01:14, 1.19it/s]
Loading 0: 76%|███████▌ | 275/363 [03:12<01:32, 1.05s/it]
Loading 0: 76%|███████▋ | 277/363 [03:12<00:54, 1.57it/s]
Loading 0: 77%|███████▋ | 278/363 [03:13<00:50, 1.69it/s]
Loading 0: 77%|███████▋ | 279/363 [03:13<00:47, 1.78it/s]
Loading 0: 77%|███████▋ | 280/363 [03:13<00:38, 2.18it/s]
Loading 0: 78%|███████▊ | 282/363 [03:15<00:49, 1.65it/s]
Loading 0: 78%|███████▊ | 283/363 [03:16<01:07, 1.19it/s]
Loading 0: 78%|███████▊ | 284/363 [03:18<01:23, 1.06s/it]
Loading 0: 79%|███████▉ | 286/363 [03:18<00:49, 1.57it/s]
Loading 0: 79%|███████▉ | 287/363 [03:19<00:45, 1.67it/s]
Loading 0: 79%|███████▉ | 288/363 [03:19<00:42, 1.77it/s]
Loading 0: 80%|███████▉ | 289/363 [03:19<00:34, 2.17it/s]
Loading 0: 80%|████████ | 291/363 [03:21<00:43, 1.64it/s]
Loading 0: 80%|████████ | 292/363 [03:23<01:00, 1.18it/s]
Loading 0: 81%|████████ | 293/363 [03:24<01:13, 1.06s/it]
Loading 0: 81%|████████▏ | 295/363 [03:24<00:43, 1.57it/s]
Loading 0: 82%|████████▏ | 296/363 [03:25<00:39, 1.69it/s]
Loading 0: 82%|████████▏ | 297/363 [03:25<00:37, 1.78it/s]
Loading 0: 82%|████████▏ | 298/363 [03:26<00:30, 2.16it/s]
Loading 0: 82%|████████▏ | 299/363 [03:27<00:49, 1.29it/s]
Loading 0: 83%|████████▎ | 301/363 [03:28<00:34, 1.81it/s]
Loading 0: 83%|████████▎ | 302/363 [03:28<00:32, 1.88it/s]
Loading 0: 83%|████████▎ | 303/363 [03:28<00:26, 2.28it/s]
Loading 0: 84%|████████▎ | 304/363 [03:29<00:24, 2.45it/s]
Loading 0: 84%|████████▍ | 306/363 [03:30<00:32, 1.73it/s]
Loading 0: 85%|████████▍ | 307/363 [03:32<00:46, 1.20it/s]
Loading 0: 85%|████████▌ | 310/363 [03:33<00:36, 1.46it/s]
Loading 0: 86%|████████▌ | 311/363 [03:35<00:45, 1.16it/s]
Loading 0: 86%|████████▌ | 312/363 [03:37<00:53, 1.05s/it]
Loading 0: 87%|████████▋ | 314/363 [03:37<00:32, 1.50it/s]
Loading 0: 87%|████████▋ | 315/363 [03:37<00:29, 1.61it/s]
Loading 0: 87%|████████▋ | 316/363 [03:38<00:27, 1.71it/s]
Loading 0: 87%|████████▋ | 317/363 [03:38<00:21, 2.09it/s]
Loading 0: 88%|████████▊ | 319/363 [03:40<00:26, 1.63it/s]
Loading 0: 88%|████████▊ | 320/363 [03:41<00:36, 1.19it/s]
Loading 0: 88%|████████▊ | 321/363 [03:43<00:44, 1.05s/it]
Loading 0: 89%|████████▉ | 323/363 [03:43<00:25, 1.57it/s]
Loading 0: 89%|████████▉ | 324/363 [03:43<00:23, 1.68it/s]
Loading 0: 90%|████████▉ | 325/363 [03:44<00:21, 1.78it/s]
Loading 0: 90%|████████▉ | 326/363 [03:44<00:16, 2.18it/s]
Loading 0: 90%|█████████ | 328/363 [03:46<00:21, 1.66it/s]
Loading 0: 91%|█████████ | 329/363 [03:47<00:28, 1.20it/s]
Loading 0: 91%|█████████ | 330/363 [03:49<00:34, 1.04s/it]
Loading 0: 91%|█████████▏| 332/363 [03:49<00:19, 1.58it/s]
Loading 0: 92%|█████████▏| 333/363 [03:49<00:17, 1.69it/s]
Loading 0: 92%|█████████▏| 334/363 [03:50<00:16, 1.78it/s]
Loading 0: 92%|█████████▏| 335/363 [03:50<00:12, 2.18it/s]
Loading 0: 93%|█████████▎| 337/363 [03:52<00:15, 1.66it/s]
Loading 0: 93%|█████████▎| 338/363 [03:53<00:21, 1.19it/s]
Loading 0: 93%|█████████▎| 339/363 [03:55<00:25, 1.05s/it]
Loading 0: 94%|█████████▍| 341/363 [03:55<00:13, 1.57it/s]
Loading 0: 94%|█████████▍| 342/363 [03:56<00:12, 1.69it/s]
Loading 0: 94%|█████████▍| 343/363 [03:56<00:11, 1.79it/s]
Loading 0: 95%|█████████▍| 344/363 [03:56<00:08, 2.18it/s]
Loading 0: 95%|█████████▌| 346/363 [03:58<00:10, 1.65it/s]
Loading 0: 96%|█████████▌| 347/363 [03:59<00:13, 1.19it/s]
Loading 0: 96%|█████████▌| 348/363 [04:01<00:15, 1.05s/it]
Loading 0: 96%|█████████▋| 350/363 [04:01<00:08, 1.58it/s]
Loading 0: 97%|█████████▋| 351/363 [04:02<00:07, 1.70it/s]
Loading 0: 97%|█████████▋| 352/363 [04:02<00:06, 1.79it/s]
Loading 0: 97%|█████████▋| 353/363 [04:02<00:04, 2.19it/s]
Loading 0: 98%|█████████▊| 355/363 [04:04<00:04, 1.65it/s]
Loading 0: 98%|█████████▊| 356/363 [04:06<00:05, 1.19it/s]
Loading 0: 98%|█████████▊| 357/363 [04:07<00:06, 1.05s/it]
Loading 0: 99%|█████████▉| 359/363 [04:07<00:02, 1.57it/s]
Loading 0: 99%|█████████▉| 360/363 [04:08<00:01, 1.68it/s]
Loading 0: 99%|█████████▉| 361/363 [04:08<00:01, 1.79it/s]
Loading 0: 100%|█████████▉| 362/363 [04:08<00:00, 2.19it/s]
Job chaiml-nitral-dpos1-beta-5231-v7-mkmlizer completed after 331.02s with status: succeeded
Stopping job with name chaiml-nitral-dpos1-beta-5231-v7-mkmlizer
Pipeline stage MKMLizer completed in 333.08s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.77s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nitral-dpos1-beta-5231-v7
Waiting for inference service chaiml-nitral-dpos1-beta-5231-v7 to be ready
chaiml-cogito32b-dpos2-39363-v8-mkmlizer:
Loading 0: 0%| | 0/771 [00:00<?, ?it/s]
Loading 0: 0%| | 3/771 [00:03<13:14, 1.03s/it]
Loading 0: 1%| | 4/771 [00:06<21:51, 1.71s/it]
Loading 0: 1%| | 5/771 [00:09<27:51, 2.18s/it]
Loading 0: 1%| | 8/771 [00:09<12:04, 1.05it/s]
Loading 0: 1%|▏ | 10/771 [00:10<09:06, 1.39it/s]
Loading 0: 1%|▏ | 11/771 [00:10<08:45, 1.45it/s]
Loading 0: 2%|▏ | 13/771 [00:10<06:07, 2.06it/s]
Loading 0: 2%|▏ | 15/771 [00:14<10:33, 1.19it/s]
Loading 0: 2%|▏ | 16/771 [00:16<14:24, 1.15s/it]
Loading 0: 2%|▏ | 17/771 [00:18<18:10, 1.45s/it]
Loading 0: 3%|▎ | 20/771 [00:19<09:32, 1.31it/s]
Loading 0: 3%|▎ | 22/771 [00:19<07:40, 1.63it/s]
Loading 0: 3%|▎ | 23/771 [00:20<07:31, 1.66it/s]
Loading 0: 3%|▎ | 25/771 [00:20<05:25, 2.29it/s]
Loading 0: 4%|▎ | 27/771 [00:23<09:59, 1.24it/s]
Loading 0: 4%|▎ | 28/771 [00:25<13:39, 1.10s/it]
Loading 0: 4%|▍ | 29/771 [00:28<17:18, 1.40s/it]
Loading 0: 4%|▍ | 32/771 [00:28<09:12, 1.34it/s]
Loading 0: 4%|▍ | 34/771 [00:29<07:26, 1.65it/s]
Loading 0: 5%|▍ | 35/771 [00:29<07:20, 1.67it/s]
Loading 0: 5%|▍ | 37/771 [00:29<05:20, 2.29it/s]
Loading 0: 5%|▍ | 38/771 [00:32<10:37, 1.15it/s]
Loading 0: 5%|▌ | 40/771 [00:32<07:02, 1.73it/s]
Loading 0: 5%|▌ | 41/771 [00:33<07:01, 1.73it/s]
Loading 0: 6%|▌ | 43/771 [00:33<05:42, 2.13it/s]
Loading 0: 6%|▌ | 45/771 [00:34<04:17, 2.82it/s]
Loading 0: 6%|▌ | 46/771 [00:34<03:51, 3.13it/s]
Loading 0: 6%|▌ | 47/771 [00:37<11:09, 1.08it/s]
Loading 0: 6%|▌ | 48/771 [00:40<16:21, 1.36s/it]
Loading 0: 7%|▋ | 51/771 [00:43<14:14, 1.19s/it]
Loading 0: 7%|▋ | 52/771 [00:46<18:14, 1.52s/it]
Loading 0: 7%|▋ | 53/771 [00:48<21:58, 1.84s/it]
Loading 0: 7%|▋ | 56/771 [00:49<11:33, 1.03it/s]
Loading 0: 7%|▋ | 57/771 [00:49<10:40, 1.11it/s]
Loading 0: 8%|▊ | 59/771 [00:50<08:07, 1.46it/s]
Loading 0: 8%|▊ | 61/771 [00:50<05:55, 2.00it/s]
Loading 0: 8%|▊ | 63/771 [00:53<09:50, 1.20it/s]
Loading 0: 8%|▊ | 64/771 [00:56<14:42, 1.25s/it]
Loading 0: 8%|▊ | 65/771 [00:59<19:20, 1.64s/it]
Loading 0: 9%|▉ | 68/771 [00:59<10:19, 1.13it/s]
Loading 0: 9%|▉ | 69/771 [01:00<09:38, 1.21it/s]
Loading 0: 9%|▉ | 71/771 [01:01<07:27, 1.57it/s]
Loading 0: 9%|▉ | 73/771 [01:01<05:26, 2.14it/s]
Loading 0: 10%|▉ | 75/771 [01:04<09:25, 1.23it/s]
Loading 0: 10%|▉ | 76/771 [01:07<14:16, 1.23s/it]
Loading 0: 10%|▉ | 77/771 [01:10<18:57, 1.64s/it]
Loading 0: 10%|█ | 80/771 [01:10<10:08, 1.13it/s]
Loading 0: 11%|█ | 81/771 [01:11<09:29, 1.21it/s]
Loading 0: 11%|█ | 83/771 [01:11<07:19, 1.56it/s]
Loading 0: 11%|█ | 85/771 [01:12<05:22, 2.12it/s]
Loading 0: 11%|█▏ | 87/771 [01:15<09:15, 1.23it/s]
Loading 0: 11%|█▏ | 88/771 [01:18<14:07, 1.24s/it]
Loading 0: 12%|█▏ | 89/771 [01:21<18:42, 1.65s/it]
Loading 0: 12%|█▏ | 92/771 [01:21<10:00, 1.13it/s]
Loading 0: 12%|█▏ | 93/771 [01:22<09:20, 1.21it/s]
Loading 0: 12%|█▏ | 95/771 [01:22<07:12, 1.56it/s]
Loading 0: 13%|█▎ | 97/771 [01:22<05:18, 2.12it/s]
Loading 0: 13%|█▎ | 98/771 [01:26<10:51, 1.03it/s]
Loading 0: 13%|█▎ | 100/771 [01:26<07:14, 1.55it/s]
Loading 0: 13%|█▎ | 101/771 [01:26<07:03, 1.58it/s]
Loading 0: 13%|█▎ | 103/771 [01:27<05:38, 1.98it/s]
Loading 0: 14%|█▎ | 105/771 [01:27<04:12, 2.64it/s]
Loading 0: 14%|█▎ | 106/771 [01:27<03:44, 2.96it/s]
Loading 0: 14%|█▍ | 107/771 [01:30<10:17, 1.07it/s]
Loading 0: 14%|█▍ | 108/771 [01:33<15:54, 1.44s/it]
Loading 0: 14%|█▍ | 109/771 [01:37<20:38, 1.87s/it]
Loading 0: 15%|█▍ | 112/771 [01:37<10:03, 1.09it/s]
Loading 0: 15%|█▍ | 113/771 [01:37<09:17, 1.18it/s]
Loading 0: 15%|█▍ | 115/771 [01:38<07:02, 1.55it/s]
Loading 0: 15%|█▌ | 117/771 [01:38<05:05, 2.14it/s]
Loading 0: 15%|█▌ | 119/771 [01:41<08:51, 1.23it/s]
Loading 0: 16%|█▌ | 120/771 [01:44<13:28, 1.24s/it]
Loading 0: 16%|█▌ | 121/771 [01:47<17:52, 1.65s/it]
Loading 0: 16%|█▌ | 124/771 [01:47<09:28, 1.14it/s]
Loading 0: 16%|█▋ | 126/771 [01:48<07:27, 1.44it/s]
Loading 0: 16%|█▋ | 127/771 [01:49<07:12, 1.49it/s]
Loading 0: 17%|█▋ | 129/771 [01:49<05:09, 2.07it/s]
Loading 0: 17%|█▋ | 131/771 [01:52<08:49, 1.21it/s]
Loading 0: 17%|█▋ | 132/771 [01:55<13:18, 1.25s/it]
Loading 0: 17%|█▋ | 133/771 [01:58<17:36, 1.66s/it]
Loading 0: 18%|█▊ | 137/771 [01:59<08:50, 1.19it/s]
Loading 0: 18%|█▊ | 139/771 [01:59<07:11, 1.46it/s]
Loading 0: 18%|█▊ | 141/771 [02:00<05:29, 1.91it/s]
Loading 0: 18%|█▊ | 142/771 [02:03<09:59, 1.05it/s]
Loading 0: 19%|█▊ | 144/771 [02:03<06:53, 1.52it/s]
Loading 0: 19%|█▉ | 145/771 [02:03<06:42, 1.55it/s]
Loading 0: 19%|█▉ | 147/771 [02:04<05:23, 1.93it/s]
Loading 0: 19%|█▉ | 149/771 [02:04<04:02, 2.57it/s]
Loading 0: 20%|█▉ | 151/771 [02:07<07:47, 1.33it/s]
Loading 0: 20%|█▉ | 152/771 [02:10<12:24, 1.20s/it]
Loading 0: 20%|██ | 155/771 [02:13<11:31, 1.12s/it]
Loading 0: 20%|██ | 156/771 [02:16<15:00, 1.46s/it]
Loading 0: 20%|██ | 157/771 [02:20<18:25, 1.80s/it]
Loading 0: 21%|██ | 160/771 [02:20<10:03, 1.01it/s]
Loading 0: 21%|██ | 161/771 [02:20<09:16, 1.10it/s]
Loading 0: 21%|██ | 163/771 [02:21<07:05, 1.43it/s]
Loading 0: 21%|██▏ | 165/771 [02:21<05:11, 1.95it/s]
Loading 0: 22%|██▏ | 166/771 [02:21<04:31, 2.23it/s]
Loading 0: 22%|██▏ | 167/771 [02:24<10:07, 1.01s/it]
Loading 0: 22%|██▏ | 168/771 [02:28<15:08, 1.51s/it]
Loading 0: 22%|██▏ | 171/771 [02:31<12:39, 1.27s/it]
Loading 0: 22%|██▏ | 172/771 [02:34<16:06, 1.61s/it]
Loading 0: 22%|██▏ | 173/771 [02:37<19:21, 1.94s/it]
Loading 0: 23%|██▎ | 176/771 [02:37<10:10, 1.03s/it]
Loading 0: 23%|██▎ | 177/771 [02:37<09:16, 1.07it/s]
Loading 0: 23%|██▎ | 179/771 [02:38<07:01, 1.40it/s]
Loading 0: 23%|██▎ | 181/771 [02:38<05:03, 1.94it/s]
Loading 0: 24%|██▎ | 183/771 [02:41<08:12, 1.19it/s]
Loading 0: 24%|██▍ | 184/771 [02:44<12:15, 1.25s/it]
Loading 0: 24%|██▍ | 185/771 [02:47<16:04, 1.65s/it]
Loading 0: 24%|██▍ | 188/771 [02:48<08:34, 1.13it/s]
Loading 0: 25%|██▍ | 190/771 [02:48<06:42, 1.44it/s]
Loading 0: 25%|██▍ | 191/771 [02:49<06:29, 1.49it/s]
Loading 0: 25%|██▌ | 193/771 [02:49<04:39, 2.07it/s]
Loading 0: 25%|██▌ | 195/771 [02:52<07:55, 1.21it/s]
Loading 0: 25%|██▌ | 196/771 [02:55<11:59, 1.25s/it]
Loading 0: 26%|██▌ | 197/771 [02:58<15:48, 1.65s/it]
Loading 0: 26%|██▌ | 200/771 [02:58<08:23, 1.13it/s]
Loading 0: 26%|██▌ | 201/771 [02:59<07:49, 1.21it/s]
Loading 0: 26%|██▋ | 203/771 [02:59<06:02, 1.57it/s]
Loading 0: 27%|██▋ | 205/771 [03:00<04:25, 2.13it/s]
Loading 0: 27%|██▋ | 207/771 [03:03<07:36, 1.24it/s]
Loading 0: 27%|██▋ | 208/771 [03:06<11:46, 1.25s/it]
Loading 0: 27%|██▋ | 209/771 [03:09<15:39, 1.67s/it]
Loading 0: 27%|██▋ | 212/771 [03:09<08:21, 1.11it/s]
Loading 0: 28%|██▊ | 213/771 [03:10<07:49, 1.19it/s]
Loading 0: 28%|██▊ | 215/771 [03:11<06:02, 1.53it/s]
Loading 0: 28%|██▊ | 217/771 [03:11<04:24, 2.10it/s]
Loading 0: 28%|██▊ | 218/771 [03:14<09:01, 1.02it/s]
Loading 0: 29%|██▊ | 220/771 [03:14<06:00, 1.53it/s]
Loading 0: 29%|██▊ | 221/771 [03:15<05:52, 1.56it/s]
Loading 0: 29%|██▉ | 223/771 [03:15<04:42, 1.94it/s]
Loading 0: 29%|██▉ | 225/771 [03:15<03:27, 2.63it/s]
Loading 0: 29%|██▉ | 226/771 [03:16<03:04, 2.95it/s]
Loading 0: 29%|██▉ | 227/771 [03:19<08:33, 1.06it/s]
Loading 0: 30%|██▉ | 228/771 [03:22<13:22, 1.48s/it]
Loading 0: 30%|██▉ | 231/771 [03:25<11:17, 1.26s/it]
Loading 0: 30%|███ | 232/771 [03:28<14:30, 1.61s/it]
Loading 0: 30%|███ | 233/771 [03:31<17:32, 1.96s/it]
Loading 0: 31%|███ | 236/771 [03:31<09:12, 1.03s/it]
Loading 0: 31%|███ | 237/771 [03:32<08:26, 1.05it/s]
Loading 0: 31%|███ | 239/771 [03:33<06:24, 1.38it/s]
Loading 0: 31%|███▏ | 241/771 [03:33<04:36, 1.92it/s]
Loading 0: 32%|███▏ | 243/771 [03:36<07:30, 1.17it/s]
Loading 0: 32%|███▏ | 244/771 [03:39<11:10, 1.27s/it]
Loading 0: 32%|███▏ | 245/771 [03:42<14:39, 1.67s/it]
Loading 0: 32%|███▏ | 248/771 [03:42<07:48, 1.12it/s]
Loading 0: 32%|███▏ | 250/771 [03:43<06:09, 1.41it/s]
Loading 0: 33%|███▎ | 251/771 [03:43<05:57, 1.46it/s]
Loading 0: 33%|███▎ | 253/771 [03:44<04:14, 2.03it/s]
Loading 0: 33%|███▎ | 255/771 [03:47<07:13, 1.19it/s]
Loading 0: 33%|███▎ | 256/771 [03:50<10:53, 1.27s/it]
Loading 0: 33%|███▎ | 257/771 [03:53<14:20, 1.67s/it]
Loading 0: 34%|███▍ | 261/771 [03:54<07:12, 1.18it/s]
Loading 0: 34%|███▍ | 263/771 [03:54<05:51, 1.44it/s]
Loading 0: 34%|███▍ | 265/771 [03:55<04:26, 1.90it/s]
Loading 0: 35%|███▍ | 267/771 [03:58<06:56, 1.21it/s]
Loading 0: 35%|███▍ | 268/771 [04:01<10:11, 1.22s/it]
Loading 0: 35%|███▍ | 269/771 [04:04<13:24, 1.60s/it]
Loading 0: 35%|███▌ | 272/771 [04:04<07:21, 1.13it/s]
Loading 0: 35%|███▌ | 273/771 [04:05<06:54, 1.20it/s]
Loading 0: 36%|███▌ | 275/771 [04:05<05:22, 1.54it/s]
Loading 0: 36%|███▌ | 277/771 [04:05<03:55, 2.10it/s]
Loading 0: 36%|███▌ | 278/771 [04:09<08:00, 1.03it/s]
Loading 0: 36%|███▋ | 280/771 [04:09<05:20, 1.53it/s]
Loading 0: 36%|███▋ | 281/771 [04:09<05:13, 1.56it/s]
Loading 0: 37%|███▋ | 283/771 [04:10<04:10, 1.95it/s]
Loading 0: 37%|███▋ | 285/771 [04:10<03:04, 2.63it/s]
Loading 0: 37%|███▋ | 286/771 [04:10<02:43, 2.96it/s]
Loading 0: 37%|███▋ | 287/771 [04:13<07:36, 1.06it/s]
Loading 0: 37%|███▋ | 288/771 [04:17<12:06, 1.50s/it]
Loading 0: 38%|███▊ | 291/771 [04:20<10:19, 1.29s/it]
Loading 0: 38%|███▊ | 292/771 [04:23<13:10, 1.65s/it]
Loading 0: 38%|███▊ | 293/771 [04:26<15:52, 1.99s/it]
Loading 0: 38%|███▊ | 296/771 [04:26<08:19, 1.05s/it]
Loading 0: 39%|███▊ | 297/771 [04:27<07:37, 1.04it/s]
Loading 0: 39%|███▉ | 299/771 [04:28<05:45, 1.37it/s]
Loading 0: 39%|███▉ | 301/771 [04:28<04:08, 1.89it/s]
Loading 0: 39%|███▉ | 303/771 [04:31<06:44, 1.16it/s]
Loading 0: 39%|███▉ | 304/771 [04:34<10:02, 1.29s/it]
Loading 0: 40%|███▉ | 305/771 [04:37<13:10, 1.70s/it]
Loading 0: 40%|███▉ | 308/771 [04:37<07:00, 1.10it/s]
Loading 0: 40%|████ | 310/771 [04:38<05:30, 1.39it/s]
Loading 0: 40%|████ | 311/771 [04:39<05:19, 1.44it/s]
Loading 0: 41%|████ | 313/771 [04:39<03:47, 2.02it/s]
Loading 0: 41%|████ | 315/771 [04:42<06:27, 1.18it/s]
Loading 0: 41%|████ | 316/771 [04:45<09:44, 1.28s/it]
Loading 0: 41%|████ | 317/771 [04:48<12:51, 1.70s/it]
Loading 0: 42%|████▏ | 320/771 [04:48<06:48, 1.10it/s]
Loading 0: 42%|████▏ | 321/771 [04:49<06:21, 1.18it/s]
Loading 0: 42%|████▏ | 323/771 [04:50<04:54, 1.52it/s]
Loading 0: 42%|████▏ | 325/771 [04:50<03:33, 2.09it/s]
Loading 0: 42%|████▏ | 327/771 [04:53<06:08, 1.20it/s]
Loading 0: 43%|████▎ | 328/771 [04:56<09:20, 1.26s/it]
Loading 0: 43%|████▎ | 329/771 [04:59<12:21, 1.68s/it]
Loading 0: 43%|████▎ | 332/771 [05:00<06:34, 1.11it/s]
Loading 0: 43%|████▎ | 334/771 [05:00<05:11, 1.40it/s]
Loading 0: 43%|████▎ | 335/771 [05:01<05:00, 1.45it/s]
Loading 0: 44%|████▎ | 337/771 [05:01<03:34, 2.02it/s]
Loading 0: 44%|████▍ | 338/771 [05:04<07:19, 1.01s/it]
Loading 0: 44%|████▍ | 340/771 [05:04<04:48, 1.49it/s]
Loading 0: 44%|████▍ | 341/771 [05:05<04:41, 1.53it/s]
Loading 0: 44%|████▍ | 343/771 [05:05<03:43, 1.92it/s]
Loading 0: 45%|████▍ | 345/771 [05:06<02:42, 2.62it/s]
Loading 0: 45%|████▍ | 346/771 [05:06<02:24, 2.94it/s]
Loading 0: 45%|████▌ | 347/771 [05:09<06:47, 1.04it/s]
Loading 0: 45%|████▌ | 348/771 [05:12<10:35, 1.50s/it]
Loading 0: 46%|████▌ | 351/771 [05:15<08:54, 1.27s/it]
Loading 0: 46%|████▌ | 352/771 [05:18<11:27, 1.64s/it]
Loading 0: 46%|████▌ | 353/771 [05:22<13:49, 1.98s/it]
Loading 0: 46%|████▌ | 356/771 [05:22<07:14, 1.05s/it]
Loading 0: 46%|████▋ | 357/771 [05:22<06:37, 1.04it/s]
Loading 0: 47%|████▋ | 359/771 [05:23<05:00, 1.37it/s]
Loading 0: 47%|████▋ | 361/771 [05:23<03:36, 1.90it/s]
Loading 0: 47%|████▋ | 363/771 [05:26<05:52, 1.16it/s]
Loading 0: 47%|████▋ | 364/771 [05:30<08:45, 1.29s/it]
Loading 0: 47%|████▋ | 365/771 [05:33<11:29, 1.70s/it]
Loading 0: 48%|████▊ | 368/771 [05:33<06:07, 1.10it/s]
Loading 0: 48%|████▊ | 369/771 [05:33<05:42, 1.17it/s]
Loading 0: 48%|████▊ | 371/771 [05:34<04:23, 1.52it/s]
Loading 0: 48%|████▊ | 373/771 [05:34<03:11, 2.08it/s]
Loading 0: 49%|████▊ | 375/771 [05:37<05:30, 1.20it/s]
Loading 0: 49%|████▉ | 376/771 [05:41<08:36, 1.31s/it]
Loading 0: 49%|████▉ | 377/771 [05:44<11:35, 1.76s/it]
Loading 0: 49%|████▉ | 380/771 [05:44<06:10, 1.05it/s]
Loading 0: 49%|████▉ | 381/771 [05:45<05:46, 1.13it/s]
Loading 0: 50%|████▉ | 383/771 [05:46<04:26, 1.46it/s]
Loading 0: 50%|████▉ | 385/771 [05:46<03:13, 1.99it/s]
Loading 0: 50%|█████ | 387/771 [05:49<05:22, 1.19it/s]
Loading 0: 50%|█████ | 388/771 [05:52<08:05, 1.27s/it]
Loading 0: 50%|█████ | 389/771 [05:55<10:37, 1.67s/it]
Loading 0: 51%|█████ | 392/771 [05:55<05:39, 1.12it/s]
Loading 0: 51%|█████ | 393/771 [05:56<05:17, 1.19it/s]
Loading 0: 51%|█████ | 395/771 [05:57<04:05, 1.53it/s]
Loading 0: 51%|█████▏ | 397/771 [05:57<02:57, 2.10it/s]
Loading 0: 52%|█████▏ | 398/771 [06:00<06:04, 1.02it/s]
Loading 0: 52%|█████▏ | 400/771 [06:00<04:02, 1.53it/s]
Loading 0: 52%|█████▏ | 401/771 [06:01<03:56, 1.57it/s]
Loading 0: 52%|█████▏ | 403/771 [06:01<03:08, 1.95it/s]
Loading 0: 53%|█████▎ | 405/771 [06:02<02:17, 2.66it/s]
Loading 0: 53%|█████▎ | 406/771 [06:02<02:01, 3.01it/s]
Loading 0: 53%|█████▎ | 407/771 [06:05<05:42, 1.06it/s]
Loading 0: 53%|█████▎ | 408/771 [06:08<08:55, 1.48s/it]
Loading 0: 53%|█████▎ | 411/771 [06:11<07:31, 1.25s/it]
Loading 0:
chaiml-mistral32-deeps-89261-v11-mkmlizer: quantized model in 521.462s
chaiml-mistral32-deeps-89261-v11-mkmlizer: Processed model ChaiML/mistral32_deepseek_cpos3_cpos21322_sfts11306_seq96_exp6 in 576.573s
chaiml-mistral32-deeps-89261-v11-mkmlizer: creating bucket guanaco-mkml-models
chaiml-mistral32-deeps-89261-v11-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-mistral32-deeps-89261-v11-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia
chaiml-mistral32-deeps-89261-v11-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia/config.json
chaiml-mistral32-deeps-89261-v11-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia/special_tokens_map.json
chaiml-mistral32-deeps-89261-v11-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia/tokenizer_config.json
chaiml-mistral32-deeps-89261-v11-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral32-deeps-89261-v11/nvidia/tokenizer.json
Job chaiml-mistral32-deeps-89261-v11-mkmlizer completed after 609.87s with status: succeeded
Stopping job with name chaiml-mistral32-deeps-89261-v11-mkmlizer
Pipeline stage MKMLizer completed in 612.52s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.84s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-mistral32-deeps-89261-v11
Waiting for inference service chaiml-mistral32-deeps-89261-v11 to be ready
Inference service chaiml-nitral-dpos1-beta-5231-v7 ready after 303.3506917953491s
Pipeline stage MKMLDeployer completed in 307.32s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 4.129997968673706s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.932973861694336s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.566399097442627s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.5381314754486084s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 3.441862106323242s
5 requests
0 failed requests
5th percentile: 2.543784999847412
10th percentile: 2.5494385242462156
20th percentile: 2.5607455730438233
30th percentile: 2.6397140502929686
40th percentile: 2.7863439559936523
50th percentile: 2.932973861694336
60th percentile: 3.1365291595458986
70th percentile: 3.340084457397461
80th percentile: 3.5794892787933352
90th percentile: 3.8547436237335204
95th percentile: 3.992370796203613
99th percentile: 4.102472534179688
chaiml-mistral31-24b-s-69496-v24-mkmlizer: quantized model in 526.909s
mean time: 3.121872901916504
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Processed model ChaiML/mistral31-24b-simpoexp1-s1-new-sft-retryv2top20lex-2e in 607.278s
Pipeline stage StressChecker completed in 32.52s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: creating bucket guanaco-mkml-models
run pipeline stage %s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
chaiml-mistral31-24b-s-69496-v24-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.83s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/config.json
run pipeline stage %s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/special_tokens_map.json
Running pipeline stage TriggerMKMLProfilingPipeline
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/tokenizer_config.json
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.87s
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/tokenizer.json
Shutdown handler de-registered
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/flywheel_model.1.safetensors
chaiml-mistral31-24b-s-69496-v24-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-mistral31-24b-s-69496-v24/nvidia/flywheel_model.1.safetensors
chaiml-nitral-dpos1-beta_5231_v7 status is now deployed due to DeploymentManager action
chaiml-nitral-dpos1-beta_5231_v7 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e4d227a5-7315-4b2c-b58c-65d3531fa05a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:16:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7ac5f0e2-238a-4eab-90a9-56813dc1902c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:20:43 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'06725311-1c05-41dd-96d4-0f4d892c3710\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:25:46 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'86836daf-12a9-4f45-9242-ccf5a04b7d83\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:30:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3727f194-9483-4ee4-80fc-c9579a6d75f5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:35:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4e42815a-8e1b-4782-9d4e-7c9c050a338c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:40:41 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1f5c1e85-8f53-4880-809f-100385fc78a1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:45:50 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'252fb268-78c1-4c47-a279-36e62ae5a78f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:50:49 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'db3548d0-902f-45f4-8439-f03117fc843f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 00:55:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2b4d8eab-19c4-4ed7-947e-43ab320d724d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:00:49 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cf83aa44-bfe9-498f-8dac-77059202f239\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:05:49 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8d3b4b47-ac65-431d-941c-18ec6f2bd7ba\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:10:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d51b15c8-c4d6-4b7a-81f9-4e0e6460a64c\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:16:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bcfadace-6c6c-4a61-a4dd-b5060cf4fac4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:21:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fe27d3d2-69c5-4613-ae77-50776d67f0b3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:25:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f6571818-e249-4ab4-b451-217bd38c64d7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:31:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4eaf5e6a-92e8-4d63-bd4e-e907862d9d21\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:35:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1f1dafe8-e285-4a4b-a88f-ee4d480ad7b9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:40:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7a73b596-df12-4d07-b377-8ee36da5b6ae\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:45:53 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'14edaa6b-dfd6-402d-b6ff-ed6dce836eb0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:50:55 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'788461f6-4454-42c3-aced-3ec15215e70d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 01:55:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'64c9fe4f-124d-42e7-9a0d-414aed717a2b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:01:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1aa7da9a-e087-499d-8d1a-f8c7d755d9c9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:05:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5c0dfdb5-ab72-4df4-9d6b-9159e230d6f9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:11:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bf3b0a7e-5a22-455d-a95c-a9ccd0a662ae\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:16:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cdcccfb9-9142-4795-b29b-d51cb436b3cc\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:21:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'07335bc6-f153-4479-9c75-130c73af4425\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:26:05 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7fdf1636-f878-4f9c-89f3-eba3c5d4bd65\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:31:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e9983536-2fdc-4566-b797-f37f06ce4ab4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:36:06 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dc245625-d632-40e6-86b4-e299c0194fa9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:41:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'823add2a-0ac6-4963-9d31-ffbeea4be1e7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:46:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'da4465c1-c80f-4520-8b96-b9dcd72e6b52\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:50:58 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'49d20e13-b6e2-4f48-b5a1-58f0ed5b2dfa\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 02:56:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e86b8e47-d0c6-4344-afbe-498d48ff6bd0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:01:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'77bd8c88-ec07-4fbd-8182-3be80562d64f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:06:01 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5c813464-40a5-4cb1-9206-5567c8ed68e8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:11:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
HTTP Request: %s %s "%s %d %s"
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4679eb77-13df-4e2f-8e51-1e5e39ae529d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:16:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b02efd47-9250-4471-b363-8f9f77b2d759\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:21:07 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4674afee-6b39-4ea2-b1e6-093f49e86d66\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:26:05 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c36247b1-4ea1-497a-badd-a1c32cc70176\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:31:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'db64760f-efac-4ffa-a4d3-0d5bf164002d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:36:06 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fb8e55f4-39bc-4a44-ba92-8d43d68bfe39\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:41:07 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4486e7f5-219c-4b26-b200-64bd0e117f11\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:46:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3d553900-5da2-4d64-8773-0a23df636efc\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:51:12 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4781299a-a9aa-4a1a-8131-665c7fd8f915\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 03:56:12 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'50700e41-a7df-4e7b-a737-f825ba1392af\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:01:09 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ba8c71bd-1dca-4c63-b8db-c39fdc344b56\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:06:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'63c7d076-fe32-48f2-944f-87b093830643\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:11:12 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ecd6c5a4-3a47-4899-9233-1675ef08605b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:16:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'97a22de5-2e87-4bcc-96c7-f27c80e24ea8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:21:13 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'83355cca-28c2-4d86-95b9-a553bafe31fc\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:26:14 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'fcfef751-0434-43c0-a7b9-289aaee7a681\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:31:12 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'aec12d54-a0d1-4c1c-8a53-c52545a1df59\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:36:15 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b2c41d2a-7929-474a-91b0-0b94e17c9b69\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:41:13 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a22e5f8e-ac51-4b36-942c-545dde5d0e3b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:46:10 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1d695dab-6989-4788-8660-8997d09c618a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:51:19 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2b751d74-fc97-45db-b3ad-ede4513a1a44\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 04:56:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d9c931a0-d1b4-426c-866f-956276857357\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:01:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b6cf1f38-748e-4b62-8702-1841fe7539a3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:06:18 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b9a3019a-b168-44f8-be7f-175232f61249\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:11:18 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ed6315b2-3ff4-465b-941c-946a561bdf89\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:16:23 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f696f6d0-7108-4b0e-a551-12e71f215e5e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:21:14 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cda60a30-844a-4fa9-9091-e0074ae1d61e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:26:16 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'0eb83245-b725-49fa-ae14-53579a00d9b2\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:31:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'320d7f06-5c91-4b48-ac04-53552188b4fc\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:36:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'63e3f2d1-d0cf-4c5c-8cba-ceb7ea8ea70d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:41:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'74a0bc7f-ba76-46ed-abfe-542001b4a031\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:46:21 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5e0fa41b-ae7e-4e43-ae4d-d54336004431\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:51:22 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3c9e25ae-34b5-4006-a52b-821d6fcd7e17\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 05:56:24 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a3e964a4-bbf0-4f21-828b-c4518c400628\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:01:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cb096a4e-166a-4ffd-a442-35f99f942395\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:06:34 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'90c0369a-63c3-4ff3-b4cd-32f3ce7cb366\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:11:26 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2fc19f6f-b53c-4f88-838b-1d74ba7e6785\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:16:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4da3554d-fe87-430f-b36a-8b236344113d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:21:29 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'feadbf12-4418-4763-b449-933c46d68dc5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:26:25 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'cbca523b-d59b-40f5-b327-93c066c4f482\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:31:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f94a44d8-fbdf-4e80-8cc8-0e25c06dbe83\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:36:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'2118899d-d384-499a-a164-f2aa09594aa9\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:41:35 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4bb87943-32dc-4917-995a-8d387c93c3c3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:46:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'169f7a15-167a-4ce9-b84e-4a029b3dfa3f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:51:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a2b823f0-7eb8-4cea-bc94-2d80e3e781ed\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 06:56:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'74fbfbad-401f-4eff-9e4c-f571bda8b57f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:01:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'091ef340-65e4-4287-8045-d25c5bfeb8da\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:06:31 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'45c60887-6571-4974-88f6-362c881ed744\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:11:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7b19fa4d-34dd-4a3e-bfb5-0b19960a7685\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:16:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a0ad5f95-dfa6-4ebc-a1ff-9626785fde97\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:21:36 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'6c1464b0-7c67-4f5a-849a-a2d55e8da087\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:26:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8e62f7d5-7247-4fec-9b2a-6ec17a085101\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:31:54 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ef03a199-51ea-4a1a-97c7-9b47eb6e8c1b\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:37:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5a565ae9-b046-4600-ba73-2b5f900f8049\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:41:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7b0ee1c3-973f-411c-80f5-f13f76509af7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:46:42 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'27959d2b-c796-4af1-a79a-4a9206767aa4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:51:42 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e368f105-50a6-4b53-84ca-cbd0cfdb3cba\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 07:56:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'43304ef2-6aa7-4a65-b5d6-f86627120f3f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:01:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'671681f2-5af6-4eaa-a3a6-82b722ef533a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:06:53 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8e269f4f-90a8-4519-be09-0abb70da9c8d\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:11:55 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3bf193fa-41df-4156-8b91-9e51da95ddd4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:16:56 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'5c515a14-47de-4845-9d4a-34ee40eee352\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:22:04 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'d4211d9e-ff23-4269-96b8-7e02a483db50\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:27:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'476397bd-5211-42ca-98d9-490e1bd5f010\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:32:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'246491b6-2b97-457a-b76c-02fb2ca1b77e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:37:02 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8e8b7478-11f2-4138-aaa1-aad3adcf9445\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:42:05 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a8bf6b75-ee04-4db0-b554-8591d4eb6af5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:51:52 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f65d188c-6f7e-47d7-b171-f644ee53dc05\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 08:57:06 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4ba39395-be27-4109-af40-b2cd9ba793f7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:02:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e6f1ea1e-36df-4519-a0cb-2b1a97893790\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:07:38 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a1f82a56-2de3-4161-9a17-83ad0dcef2a7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:16:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'372b643d-90f6-4847-a88a-5aabe2247400\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:22:17 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f5360df2-a299-4ba9-8be5-dee0ebacb718\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:27:44 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'568d80c2-d4e1-4afa-a08b-bb13a043d6af\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:36:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'f83e2459-8929-473a-bf87-5dbbac6bfed4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:42:20 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4cbdbf52-91f7-404a-83f4-7ea49f5bb625\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:47:55 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'02a9b860-55db-495c-a5a3-0345b78f4377\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 09:56:52 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'551d9c6f-da46-445d-a512-89f3febfde51\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:02:27 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'57d9075f-ebd8-4d69-aae1-c594549620ca\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:08:17 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'98d315c3-6703-4914-9b47-c479e27670fe\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:16:57 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'640473b0-60b0-4b25-ba91-485ba4d92729\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:22:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Failed to get response for submission blend_rurun_2025-08-28: ('http://guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com/models/chaiml-exp-grpo-cp312-_36146_v14/predict', '{"detail":"list index out of range"}')
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'991826fc-a019-43ee-96ab-bf07e76ef085\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:32:09 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9efa6321-4217-4a34-a10d-c263e0001d46\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:37:49 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'c8ddd20d-e2a3-45b7-a089-de201a07d0bc\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:46:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
HTTP Request: %s %s "%s %d %s"
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ed3dc50b-e302-469d-8749-40c17d08b807\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 10:52:43 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'05f5bd3c-1552-465d-ae43-01be6e79263a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:02:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
HTTP Request: %s %s "%s %d %s"
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
HTTP Request: %s %s "%s %d %s"
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'705e3529-5786-4511-adfa-d46f72e75132\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:07:51 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dde47943-ecdb-4b70-a5f0-1ad40aebb251\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:17:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'197f26e7-a11e-4862-b4d3-273664fbea64\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:22:59 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e1027317-9520-4bfa-91b9-1350b9510018\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:32:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ae1b04b8-48cb-4c37-bd01-450cb9231df3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:38:00 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1ca25a09-9fcd-4c79-9934-b62941e32104\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:47:09 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'49e3255a-41fa-4120-81f1-1c9d8dbd6213\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 11:53:03 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'af40ad71-28b4-4bb9-ba83-93e2c1bf0079\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:02:08 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'4e6b68a2-ba62-4c3c-a1a7-2a18fe51c4f7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:08:10 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b50e7309-e66a-4f5a-8ad7-ff7a9fefe7c3\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:17:10 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3d1eb620-6d30-4bd4-9dbb-7456e17acb58\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:23:13 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9ee2b567-1f0d-4651-a695-4041a7532f17\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:32:15 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1af9dc73-d42d-4e3d-b3fa-6f814a5dfdd8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:38:16 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'a3ab8ed2-944a-47f8-8374-b916992db7af\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:47:16 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'85a68359-57f3-489a-a74b-b6284ece4cc2\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 12:53:06 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'344f40ed-b91b-429f-937a-27b5b2f6c002\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:02:18 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1bfe8343-dedb-45c5-9c85-57fe23a93a3e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:08:10 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'9176cdc7-9bb5-499b-8bf7-c9f0bc88742f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:17:15 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'92d1e539-2f90-4d71-b340-264ab6524dc2\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:23:21 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'e2339855-2131-4b1e-9720-646dd513fba1\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:32:19 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'01cf3eff-0252-4ec7-a6c4-776728980602\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:38:30 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'7969e224-222b-4528-9883-0a9bc55321f0\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:47:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'dff7e82c-3011-495b-90ec-fd983ce67933\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:53:28 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'316ebb23-7d7c-437a-8058-99d1d579f808\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 13:58:39 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'37a02727-de85-4cbb-af7b-808fc2bf40f4\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:07:46 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'14a9320e-3de4-4875-8ba8-8cf01f958f0f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:13:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3cd67616-f388-4990-840d-a61b21b5ee0e\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:18:54 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'1974ef61-84aa-441a-81ee-b2e053e6a202\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:27:32 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'b70494a0-ac35-4425-8817-f3a0cc4678b8\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:33:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ab9b423b-3d4f-47c8-8f47-a6a775c03473\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:38:37 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'062c46f4-a84c-4c3f-97ea-4e4ad8fad80a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:47:44 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'23ab5f39-7354-4b11-ac08-b03517410bf5\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 14:53:43 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3a4f19ad-9346-4fef-9002-4b48a35670bb\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 15:22:34 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'639a4115-3579-43e7-aaed-57fd74d77cc7\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 15:28:42 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'8be1cd30-7cf2-4d9c-a670-2ce692af844a\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 16:03:33 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered
admin requested tearing down of chaiml-nitral-dpos1-beta_5231_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError('401\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'bd764a05-99e0-42fd-8d51-c84cedd44785\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'Date\': \'Sat, 06 Sep 2025 16:18:11 GMT\', \'Content-Length\': \'129\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\\n\'\nOriginal traceback: \n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n api_response = self.client.call_api(\n ^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n ^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\n File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n raise ApiException(http_resp=r)\n')
Shutdown handler de-registered