Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-0c1d-57904-v4-uploader
Waiting for job on chaiml-reward-dpo-0c1d-57904-v4-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-d03d-51775-v4-uploader
Waiting for job on chaiml-reward-dpo-d03d-51775-v4-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-83165-v24-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v24-uploader to finish
Stopping job with name chaiml-kimid-v9-opusdv-83165-v24-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v9-opusdv-83165-v24-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v24-uploader to finish
chaiml-reward-dpo-0c1d-57904-v4-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-0c1d-57904-v4-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-0c1d-57904-v4-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ https://mk1.ai ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ The license key for the current software has been verified as ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ belonging to: ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ Chai Research Corp. ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ║
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-0c1d-57904-v4-uploader: ║ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-0c1d-57904-v4-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ https://mk1.ai ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ belonging to: ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-reward-dpo-d03d-51775-v4-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ https://mk1.ai ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ belonging to: ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v9-opusdv-83165-v24-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v9-opusdv-83165-v24-uploader: bash: no job control in this shell
Job chaiml-kimid-v9-opusdv-83165-v24-uploader completed after 31.3s with status: failed
Stopping job with name chaiml-kimid-v9-opusdv-83165-v24-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v9-opusdv-83165-v24-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v24-uploader to finish
chaiml-kimid-v9-opusdv-83165-v24-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v9-opusdv-83165-v24-uploader: bash: no job control in this shell
chaiml-kimid-v9-opusdv-83165-v24-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v9-opusdv-83165-v24-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v9-opusdv-83165-v24-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ https://mk1.ai ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ belonging to: ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v24-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-reward-dpo-0c1d-57904-v4-uploader: Downloaded to shared memory in 154.871s
chaiml-reward-dpo-d03d-51775-v4-uploader: Downloaded to shared memory in 157.461s
chaiml-kimid-v9-opusdv-83165-v24-uploader: Downloaded to shared memory in 146.836s
chaiml-reward-dpo-d03d-51775-v4-uploader: Processed model ChaiML/reward-dpo-d03d-chaiml-kimid-v9-opusdv_83165_v12-int4-mixed in 214.963s
chaiml-reward-dpo-d03d-51775-v4-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-0c1d-57904-v4-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-0c1d-57904-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-0c1d-57904-v4-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/added_tokens.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/chat_template.jinja
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/.gitattributes
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/special_tokens_map.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/generation_config.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/tokenizer_config.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/quantization_config.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/config.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/vocab.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/merges.txt
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model.safetensors.index.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/tokenizer.json
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/merges.txt
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model.safetensors.index.json
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/tokenizer.json
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/vocab.json
chaiml-kimid-v9-opusdv-83165-v24-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-int4-mixed in 198.755s
chaiml-kimid-v9-opusdv-83165-v24-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-83165-v24-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-83165-v24-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/chat_template.jinja
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/.gitattributes
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/generation_config.json
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/added_tokens.json
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/README.md
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/tokenizer_config.json
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/special_tokens_map.json
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/config.json
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/quantization_config.json
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model.safetensors.index.json
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/tokenizer.json
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-reward-dpo-0c1d-57904-v4-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00009-of-00027.safetensors?partNumber=12&uploadId=2~V0xU3nLZ2qSBN9Z36Poj6x41wZnx318": write tcp 10.0.5.172:60770->216.153.53.63:443: write: connection reset by peer
chaiml-reward-dpo-0c1d-57904-v4-uploader: ERROR "cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00009-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~V0xU3nLZ2qSBN9Z36Poj6x41wZnx318 caused by: SignatureDoesNotMatch: status code: 403, request id: tx00000eec094258f7dca5f-00697be097-149d0ff886-default, host id:
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-reward-dpo-0c1d-57904-v4-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00024-of-00027.safetensors?partNumber=4&uploadId=2~oGAPa9vQa4EvBSOPibzJ_J-Vfax7ogg": write tcp 10.0.5.172:39530->216.153.53.63:443: write: connection reset by peer
chaiml-reward-dpo-0c1d-57904-v4-uploader: ERROR "cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00002-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~J89X720YrN2DijqSMHWmM3H9UmwdGvJ caused by: SignatureDoesNotMatch: status code: 403, request id: tx000009b0c59a723750b80-00697be099-14a26732ad-default, host id:
chaiml-reward-dpo-0c1d-57904-v4-uploader: ERROR "cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00024-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~oGAPa9vQa4EvBSOPibzJ_J-Vfax7ogg caused by: SignatureDoesNotMatch: status code: 403, request id: tx00000b848366338b76d7a-00697be0b7-14b31daae8-default, host id:
Stopping job with name chaiml-reward-dpo-0c1d-57904-v4-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-reward-dpo-0c1d-57904-v4-uploader
Waiting for job on chaiml-reward-dpo-0c1d-57904-v4-uploader to finish
chaiml-kimid-v9-opusdv-83165-v24-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v9-opusdv-83165-v24-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00024-of-00027.safetensors?partNumber=16&uploadId=2~S7QEZHVqoCGH0eqahbe59kYEDhpVtDY": write tcp 10.0.48.210:51148->216.153.53.63:443: write: connection reset by peer
chaiml-kimid-v9-opusdv-83165-v24-uploader: ERROR "cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00024-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~S7QEZHVqoCGH0eqahbe59kYEDhpVtDY caused by: SignatureDoesNotMatch: status code: 403, request id: tx0000099ea0c29c1e36b8f-00697be0f9-14a19326cd-default, host id:
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00027-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00027-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: Downloaded to shared memory in 132.546s
chaiml-reward-dpo-0c1d-57904-v4-uploader: Processed model ChaiML/reward-dpo-0c1d-chaiml-kimid-v9-opusdv_83165_v12-int4-mixed in 192.383s
chaiml-reward-dpo-0c1d-57904-v4-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-0c1d-57904-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-0c1d-57904-v4-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/.gitattributes": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/added_tokens.json": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/chat_template.jinja": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/config.json": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/generation_config.json": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/merges.txt": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model.safetensors.index.json": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/quantization_config.json": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/special_tokens_map.json": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/tokenizer.json": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/tokenizer_config.json": object size matches
chaiml-reward-dpo-0c1d-57904-v4-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/vocab.json": object size matches
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00007-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00005-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00014-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00019-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00017-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00016-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00010-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00002-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00020-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00015-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00001-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00023-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00021-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00009-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00024-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00025-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00001-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00014-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00012-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00006-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00013-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00020-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00027-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00006-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00023-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00011-of-00027.safetensors
chaiml-reward-dpo-d03d-51775-v4-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d03d-51775-v4/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00008-of-00027.safetensors
Job chaiml-reward-dpo-d03d-51775-v4-uploader completed after 698.91s with status: succeeded
Stopping job with name chaiml-reward-dpo-d03d-51775-v4-uploader
Pipeline stage VLLMUploader completed in 701.13s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.60s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-reward-dpo-d03d-51775-v4
Waiting for inference service chaiml-reward-dpo-d03d-51775-v4 to be ready
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00021-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00020-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00026-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00019-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v24-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v24/model-00022-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-83165-v24-uploader completed after 736.04s with status: failed
Stopping job with name chaiml-kimid-v9-opusdv-83165-v24-uploader
clean up pipeline due to error=VLLMUploaderError('')
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00016-of-00027.safetensors
run pipeline stage %s
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00022-of-00027.safetensors
Running pipeline stage VLLMDeleter
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00018-of-00027.safetensors
Skipping teardown as no inference service was successfully deployed
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00014-of-00027.safetensors
Pipeline stage VLLMDeleter completed in 1.16s
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00017-of-00027.safetensors
Shutdown handler de-registered
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00005-of-00027.safetensors
VLLMUploaderError('')
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv_83165_v24 status is now failed due to DeploymentManager action
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00010-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00003-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00023-of-00027.safetensors
chaiml-reward-dpo-0c1d-57904-v4-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0c1d-57904-v4/model-00009-of-00027.safetensors
Job chaiml-reward-dpo-0c1d-57904-v4-uploader completed after 478.51s with status: succeeded
Stopping job with name chaiml-reward-dpo-0c1d-57904-v4-uploader
Pipeline stage VLLMUploader completed in 806.47s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.49s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-reward-dpo-0c1d-57904-v4
Waiting for inference service chaiml-reward-dpo-0c1d-57904-v4 to be ready
Inference service chaiml-reward-dpo-d03d-51775-v4 ready after 753.1424396038055s
Pipeline stage VLLMDeployer completed in 755.16s
run pipeline stage %s
Inference service chaiml-reward-dpo-0c1d-57904-v4 ready after 652.199422121048s
Running pipeline stage StressChecker
Pipeline stage VLLMDeployer completed in 654.54s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.7770636081695557s
Received healthy response to inference request in 3.179405450820923s
Received healthy response to inference request in 2.3945791721343994s
Received healthy response to inference request in 2.2552053928375244s
Received healthy response to inference request in 2.525264024734497s
Received healthy response to inference request in 2.744842290878296s
Received healthy response to inference request in 2.291424512863159s
Received healthy response to inference request in 2.5781655311584473s
Received healthy response to inference request in 2.6847922801971436s
Received healthy response to inference request in 2.2073349952697754s
Received healthy response to inference request in 2.083087921142578s
Received healthy response to inference request in 2.6377274990081787s
Received healthy response to inference request in 2.288447856903076s
Received healthy response to inference request in 2.3858325481414795s
Received healthy response to inference request in 2.065141201019287s
Received healthy response to inference request in 2.395112991333008s
Received healthy response to inference request in 2.5645534992218018s
Received healthy response to inference request in 2.3378231525421143s
Received healthy response to inference request in 2.4205360412597656s
Received healthy response to inference request in 2.4417786598205566s
Received healthy response to inference request in 2.139525890350342s
Received healthy response to inference request in 2.0519003868103027s
Received healthy response to inference request in 2.2795369625091553s
Received healthy response to inference request in 2.276862621307373s
Received healthy response to inference request in 2.1101126670837402s
Received healthy response to inference request in 2.1550607681274414s
Received healthy response to inference request in 2.2737069129943848s
Received healthy response to inference request in 2.0651214122772217s
Received healthy response to inference request in 2.314821720123291s
Received healthy response to inference request in 2.1234357357025146s
Received healthy response to inference request in 2.023911952972412s
Received healthy response to inference request in 2.494081974029541s
Received healthy response to inference request in 3.038278818130493s
Received healthy response to inference request in 2.6372976303100586s
Received healthy response to inference request in 2.1450271606445312s
Received healthy response to inference request in 2.4883053302764893s
Received healthy response to inference request in 2.132626533508301s
Received healthy response to inference request in 2.1799850463867188s
Received healthy response to inference request in 2.8804097175598145s
Received healthy response to inference request in 2.2109696865081787s
Received healthy response to inference request in 2.1998565196990967s
Received healthy response to inference request in 2.057199716567993s
Received healthy response to inference request in 2.167811393737793s
Received healthy response to inference request in 2.2082266807556152s
Received healthy response to inference request in 2.395644187927246s
Received healthy response to inference request in 2.476539134979248s
Received healthy response to inference request in 2.626178026199341s
Received healthy response to inference request in 2.246628999710083s
Received healthy response to inference request in 2.1811790466308594s
Received healthy response to inference request in 2.222167491912842s
Received healthy response to inference request in 2.628657341003418s
Received healthy response to inference request in 2.4265999794006348s
Received healthy response to inference request in 2.2181975841522217s
Received healthy response to inference request in 2.668076992034912s
Received healthy response to inference request in 2.1781575679779053s
Received healthy response to inference request in 2.7866759300231934s
Received healthy response to inference request in 2.31491756439209s
Received healthy response to inference request in 2.238568067550659s
Received healthy response to inference request in 2.457367181777954s
Received healthy response to inference request in 2.24456524848938s
30 requests
30 requests
0 failed requests
0 failed requests
5th percentile: 2.0732172250747682
5th percentile: 2.060764479637146
10th percentile: 2.107410192489624
10th percentile: 2.117604303359985
20th percentile: 2.143926906585693
20th percentile: 2.2025783538818358
30th percentile: 2.180272603034973
30th percentile: 2.2209765195846556
40th percentile: 2.236057233810425
40th percentile: 2.2458034992218017
50th percentile: 2.2839924097061157
50th percentile: 2.3146238327026367
60th percentile: 2.347208213806152
60th percentile: 2.3947926998138427
70th percentile: 2.4895230770111083
70th percentile: 2.452206802368164
80th percentile: 2.6266738891601564
80th percentile: 2.5003183841705323
90th percentile: 2.758399033546448
90th percentile: 2.6424339771270753
95th percentile: 2.9672377228736875
95th percentile: 2.73554151058197
99th percentile: 3.1384787273406984
99th percentile: 2.7838883566856385
mean time: 2.384750811258952
mean time: 2.3559929291407267
Pipeline stage StressChecker completed in 90.85s
Pipeline stage StressChecker completed in 92.31s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.18s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.17s
Shutdown handler de-registered
Shutdown handler de-registered
Shutdown handler de-registered
chaiml-reward-dpo-0c1d-_57904_v4 status is now deployed due to DeploymentManager action
chaiml-reward-dpo-d03d-_51775_v4 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 3090.57s
Shutdown handler de-registered
chaiml-reward-dpo-0c1d-_57904_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-reward-dpo-0c1d-_57904_v4 status is now torndown due to DeploymentManager action