Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-3a2b-96782-v4-uploader
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
Waiting for job on chaiml-reward-dpo-3a2b-96782-v4-uploader to finish
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v8b-kimid-63800-v25-uploader
Waiting for job on chaiml-kimid-v8b-kimid-63800-v25-uploader to finish
chaiml-reward-dpo-483b-30936-v5-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-483b-30936-v5-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-483b-30936-v5-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ https://mk1.ai ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ The license key for the current software has been verified as ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-483b-30936-v5-uploader: ║ belonging to: ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Chai Research Corp. ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8b-kimid-63800-v25-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ https://mk1.ai ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ belonging to: ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ https://mk1.ai ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ The license key for the current software has been verified as ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ belonging to: ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-reward-dpo-3a2b-96782-v4-uploader: Downloaded to shared memory in 134.333s
chaiml-kimid-v8b-kimid-63800-v25-uploader: Downloaded to shared memory in 131.054s
chaiml-reward-dpo-483b-30936-v5-uploader: Downloaded to shared memory in 137.705s
chaiml-reward-dpo-483b-30936-v5-uploader: Processed model ChaiML/reward-dpo-483b-chaiml-kimid-v8b-kimid_63800_v12-int4-mixed in 185.436s
chaiml-reward-dpo-483b-30936-v5-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-483b-30936-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-483b-30936-v5-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/.gitattributes
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/chat_template.jinja
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/added_tokens.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/tokenizer_config.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/generation_config.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/special_tokens_map.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/quantization_config.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/config.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/merges.txt
chaiml-kimid-v8b-kimid-63800-v25-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-int4-mixed in 183.726s
chaiml-reward-dpo-3a2b-96782-v4-uploader: Processed model ChaiML/reward-dpo-3a2b-chaiml-kimid-v8b-kimid_63800_v12-int4-mixed in 188.564s
chaiml-kimid-v8b-kimid-63800-v25-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-3a2b-96782-v4-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-63800-v25-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-3a2b-96782-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-63800-v25-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25
chaiml-reward-dpo-3a2b-96782-v4-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/added_tokens.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/added_tokens.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/.gitattributes
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/chat_template.jinja
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/README.md
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/special_tokens_map.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/tokenizer_config.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/generation_config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/special_tokens_map.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/tokenizer_config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/chat_template.jinja
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/.gitattributes
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/generation_config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/quantization_config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/vocab.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/merges.txt
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model.safetensors.index.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/merges.txt
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/tokenizer.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/tokenizer.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model.safetensors.index.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model.safetensors.index.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG retryable error: ServiceUnavailable: Service Unavailable
chaiml-kimid-v8b-kimid-63800-v25-uploader: status code: 503, request id: , host id:
chaiml-kimid-v8b-kimid-63800-v25-uploader: ERROR "cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00025-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~gzEh3onpycsPJFStIPWmmgA8ngSGua2 caused by: SignatureDoesNotMatch: status code: 403, request id: tx000000ed1688af107aa07-00697a52c1-14a26732ad-default, host id:
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00027-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00027-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00027-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00024-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00017-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00014-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00021-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00006-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00011-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00026-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00015-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00012-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00022-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00020-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00002-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00010-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00025-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00019-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00005-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00001-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00009-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00016-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00018-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00020-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00014-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00012-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00010-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00015-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00011-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00004-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00013-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00007-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00003-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00022-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00004-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00015-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00008-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00026-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00005-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00006-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00014-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00023-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00003-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00021-of-00027.safetensors
Job chaiml-reward-dpo-483b-30936-v5-uploader completed after 1653.88s with status: succeeded
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00011-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00025-of-00027.safetensors
Stopping job with name chaiml-reward-dpo-483b-30936-v5-uploader
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00019-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00017-of-00027.safetensors
Pipeline stage VLLMUploader completed in 1655.85s
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00001-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00009-of-00027.safetensors
run pipeline stage %s
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00021-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00018-of-00027.safetensors
Running pipeline stage VLLMTemplater
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00012-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00005-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00004-of-00027.safetensors
Pipeline stage VLLMTemplater completed in 0.74s
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00007-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00018-of-00027.safetensors
run pipeline stage %s
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00008-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00002-of-00027.safetensors
Running pipeline stage VLLMDeployer
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00002-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00022-of-00027.safetensors
Creating inference service chaiml-reward-dpo-483b-30936-v5
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00010-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00019-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00024-of-00027.safetensors
Waiting for inference service chaiml-reward-dpo-483b-30936-v5 to be ready
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00009-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00023-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00026-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00017-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00020-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-63800-v25-uploader completed after 1663.73s with status: failed
Stopping job with name chaiml-kimid-v8b-kimid-63800-v25-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v8b-kimid-63800-v25-uploader
Waiting for job on chaiml-kimid-v8b-kimid-63800-v25-uploader to finish
Job chaiml-reward-dpo-3a2b-96782-v4-uploader completed after 1667.05s with status: succeeded
Stopping job with name chaiml-reward-dpo-3a2b-96782-v4-uploader
chaiml-kimid-v8b-kimid-63800-v25-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v8b-kimid-63800-v25-uploader: bash: no job control in this shell
Pipeline stage VLLMUploader completed in 1670.64s
chaiml-kimid-v8b-kimid-63800-v25-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
run pipeline stage %s
chaiml-kimid-v8b-kimid-63800-v25-uploader: __import__('pkg_resources').declare_namespace(__name__)
Running pipeline stage VLLMTemplater
chaiml-kimid-v8b-kimid-63800-v25-uploader: ╔═════════════════════════════════════════════════════════════════════╗
Pipeline stage VLLMTemplater completed in 0.70s
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
run pipeline stage %s
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ██████ ██████ █████ ████ ████ ║
Running pipeline stage VLLMDeployer
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
Creating inference service chaiml-reward-dpo-3a2b-96782-v4
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
Waiting for inference service chaiml-reward-dpo-3a2b-96782-v4 to be ready
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ https://mk1.ai ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ belonging to: ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8b-kimid-63800-v25-uploader: Downloaded to shared memory in 138.204s
chaiml-kimid-v8b-kimid-63800-v25-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-int4-mixed in 195.993s
chaiml-kimid-v8b-kimid-63800-v25-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-63800-v25-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-63800-v25-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/.gitattributes": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/README.md": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/added_tokens.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/chat_template.jinja": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/config.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/generation_config.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/merges.txt": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00001-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00002-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00003-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00004-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00005-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00006-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00007-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00008-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00009-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00010-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00011-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00012-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00013-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00014-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00015-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00016-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00017-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00018-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00019-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00020-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00021-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00022-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00023-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00024-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00026-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00027-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model.safetensors.index.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/quantization_config.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/special_tokens_map.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/tokenizer.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/tokenizer_config.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/vocab.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00025-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-63800-v25-uploader completed after 281.91s with status: succeeded
Stopping job with name chaiml-kimid-v8b-kimid-63800-v25-uploader
Pipeline stage VLLMUploader completed in 1949.91s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.57s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8b-kimid-63800-v25
Waiting for inference service chaiml-kimid-v8b-kimid-63800-v25 to be ready
Inference service chaiml-reward-dpo-483b-30936-v5 ready after 457.3573350906372s
Pipeline stage VLLMDeployer completed in 459.64s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.951805353164673s
Received healthy response to inference request in 2.287142276763916s
Received healthy response to inference request in 2.7931125164031982s
Received healthy response to inference request in 2.6247992515563965s
Received healthy response to inference request in 2.7661116123199463s
Received healthy response to inference request in 2.4032363891601562s
Received healthy response to inference request in 2.6112329959869385s
Received healthy response to inference request in 2.2613718509674072s
Received healthy response to inference request in 2.6068115234375s
Received healthy response to inference request in 2.716684103012085s
Received healthy response to inference request in 2.6417741775512695s
Received healthy response to inference request in 3.281804084777832s
Received healthy response to inference request in 2.491016149520874s
Received healthy response to inference request in 2.5212860107421875s
Received healthy response to inference request in 2.1873700618743896s
Received healthy response to inference request in 2.4175381660461426s
Received healthy response to inference request in 2.360775947570801s
Received healthy response to inference request in 2.2840590476989746s
Received healthy response to inference request in 2.603428840637207s
Received healthy response to inference request in 2.274653434753418s
Received healthy response to inference request in 2.350966691970825s
Received healthy response to inference request in 2.324223518371582s
Received healthy response to inference request in 2.9296908378601074s
Received healthy response to inference request in 2.344580888748169s
Received healthy response to inference request in 2.484020709991455s
Received healthy response to inference request in 2.4747493267059326s
Received healthy response to inference request in 2.294299840927124s
Received healthy response to inference request in 2.5429608821868896s
Received healthy response to inference request in 2.2925901412963867s
Received healthy response to inference request in 2.557417869567871s
30 requests
0 failed requests
5th percentile: 2.267348563671112
10th percentile: 2.283118486404419
20th percentile: 2.2939579010009767
30th percentile: 2.3490509510040285
40th percentile: 2.411817455291748
50th percentile: 2.4875184297561646
60th percentile: 2.5487436771392824
70th percentile: 2.6081379652023315
80th percentile: 2.6567561626434326
90th percentile: 2.8067703485488895
95th percentile: 2.9418538212776184
99th percentile: 3.186104452610016
mean time: 2.5227171500523884
Pipeline stage StressChecker completed in 98.08s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.59s
Shutdown handler de-registered
chaiml-reward-dpo-483b-_30936_v5 status is now deployed due to DeploymentManager action
Inference service chaiml-reward-dpo-3a2b-96782-v4 ready after 609.8824219703674s
Pipeline stage VLLMDeployer completed in 612.00s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.789708137512207s
Received healthy response to inference request in 2.2597146034240723s
Received healthy response to inference request in 2.3618903160095215s
Received healthy response to inference request in 2.534132242202759s
Received healthy response to inference request in 2.3794164657592773s
Received healthy response to inference request in 2.3253607749938965s
Received healthy response to inference request in 2.181501865386963s
Received healthy response to inference request in 2.1515183448791504s
Received healthy response to inference request in 2.2971670627593994s
Received healthy response to inference request in 2.5115163326263428s
Received healthy response to inference request in 2.344621419906616s
Received healthy response to inference request in 2.407970666885376s
Received healthy response to inference request in 2.14412784576416s
Received healthy response to inference request in 2.114715576171875s
Received healthy response to inference request in 2.437166452407837s
Received healthy response to inference request in 2.61344051361084s
Received healthy response to inference request in 2.5522654056549072s
Received healthy response to inference request in 2.2070810794830322s
Received healthy response to inference request in 2.5174131393432617s
Received healthy response to inference request in 2.3361892700195312s
Received healthy response to inference request in 2.387722969055176s
Received healthy response to inference request in 2.205249786376953s
Received healthy response to inference request in 2.2978410720825195s
Received healthy response to inference request in 2.2168450355529785s
Received healthy response to inference request in 2.17476224899292s
Received healthy response to inference request in 2.5733823776245117s
Received healthy response to inference request in 2.4147891998291016s
Received healthy response to inference request in 2.389596700668335s
Received healthy response to inference request in 2.252830743789673s
Received healthy response to inference request in 2.4442813396453857s
30 requests
0 failed requests
5th percentile: 2.1474535703659057
10th percentile: 2.172437858581543
20th percentile: 2.2067148208618166
30th percentile: 2.2576494455337524
40th percentile: 2.3143528938293456
50th percentile: 2.353255867958069
60th percentile: 2.3884724617004394
70th percentile: 2.421502375602722
80th percentile: 2.5126956939697265
90th percentile: 2.5543771028518676
95th percentile: 2.595414352416992
99th percentile: 2.738590526580811
mean time: 2.3608072996139526
Pipeline stage StressChecker completed in 86.17s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.11s
Shutdown handler de-registered
chaiml-reward-dpo-3a2b-_96782_v4 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Retrying (%r) after connection broken by '%r': %s
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 2594.97s
Shutdown handler de-registered
chaiml-reward-dpo-3a2b-_96782_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-reward-dpo-3a2b-_96782_v4 status is now protected due to ABTestQueueItem