Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-483b-30936-v5-uploader
Waiting for job on chaiml-reward-dpo-483b-30936-v5-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-3a2b-96782-v4-uploader
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
Waiting for job on chaiml-reward-dpo-3a2b-96782-v4-uploader to finish
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v8b-kimid-63800-v25-uploader
Waiting for job on chaiml-kimid-v8b-kimid-63800-v25-uploader to finish
chaiml-reward-dpo-483b-30936-v5-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-483b-30936-v5-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-483b-30936-v5-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ https://mk1.ai ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ The license key for the current software has been verified as ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-483b-30936-v5-uploader: ║ belonging to: ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Chai Research Corp. ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-483b-30936-v5-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8b-kimid-63800-v25-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ https://mk1.ai ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ belonging to: ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ https://mk1.ai ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ The license key for the current software has been verified as ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ belonging to: ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-reward-dpo-3a2b-96782-v4-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-reward-dpo-3a2b-96782-v4-uploader: Downloaded to shared memory in 134.333s
chaiml-kimid-v8b-kimid-63800-v25-uploader: Downloaded to shared memory in 131.054s
chaiml-reward-dpo-483b-30936-v5-uploader: Downloaded to shared memory in 137.705s
chaiml-reward-dpo-483b-30936-v5-uploader: Processed model ChaiML/reward-dpo-483b-chaiml-kimid-v8b-kimid_63800_v12-int4-mixed in 185.436s
chaiml-reward-dpo-483b-30936-v5-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-483b-30936-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-483b-30936-v5-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/.gitattributes
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/chat_template.jinja
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/added_tokens.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/tokenizer_config.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/generation_config.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/special_tokens_map.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/quantization_config.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/config.json
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/merges.txt
chaiml-kimid-v8b-kimid-63800-v25-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-int4-mixed in 183.726s
chaiml-reward-dpo-3a2b-96782-v4-uploader: Processed model ChaiML/reward-dpo-3a2b-chaiml-kimid-v8b-kimid_63800_v12-int4-mixed in 188.564s
chaiml-kimid-v8b-kimid-63800-v25-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-3a2b-96782-v4-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-63800-v25-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-3a2b-96782-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-63800-v25-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25
chaiml-reward-dpo-3a2b-96782-v4-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/added_tokens.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/added_tokens.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/.gitattributes
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/chat_template.jinja
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/README.md
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/special_tokens_map.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/tokenizer_config.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/generation_config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/special_tokens_map.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/tokenizer_config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/chat_template.jinja
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/.gitattributes
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/generation_config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/quantization_config.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/vocab.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/merges.txt
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model.safetensors.index.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/merges.txt
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/tokenizer.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/tokenizer.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model.safetensors.index.json
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model.safetensors.index.json
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG retryable error: ServiceUnavailable: Service Unavailable
chaiml-kimid-v8b-kimid-63800-v25-uploader: status code: 503, request id: , host id:
chaiml-kimid-v8b-kimid-63800-v25-uploader: ERROR "cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00025-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~gzEh3onpycsPJFStIPWmmgA8ngSGua2 caused by: SignatureDoesNotMatch: status code: 403, request id: tx000000ed1688af107aa07-00697a52c1-14a26732ad-default, host id:
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00027-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00027-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00027-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00024-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00017-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00014-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00021-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00006-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00011-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00026-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00015-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00012-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00022-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00020-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00002-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00010-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00025-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00019-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00005-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00001-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00009-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00016-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00018-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00020-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00014-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00012-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00010-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00015-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00011-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00004-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00013-of-00027.safetensors
chaiml-reward-dpo-483b-30936-v5-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-483b-30936-v5/model-00007-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00003-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00022-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00004-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00015-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00008-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00026-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00005-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00006-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00014-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00023-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00003-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00021-of-00027.safetensors
Job chaiml-reward-dpo-483b-30936-v5-uploader completed after 1653.88s with status: succeeded
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00011-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00025-of-00027.safetensors
Stopping job with name chaiml-reward-dpo-483b-30936-v5-uploader
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00019-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00017-of-00027.safetensors
Pipeline stage VLLMUploader completed in 1655.85s
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00001-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00009-of-00027.safetensors
run pipeline stage %s
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00021-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00018-of-00027.safetensors
Running pipeline stage VLLMTemplater
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00012-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00005-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00004-of-00027.safetensors
Pipeline stage VLLMTemplater completed in 0.74s
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00007-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00018-of-00027.safetensors
run pipeline stage %s
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00008-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00002-of-00027.safetensors
Running pipeline stage VLLMDeployer
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00002-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00022-of-00027.safetensors
Creating inference service chaiml-reward-dpo-483b-30936-v5
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00010-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00019-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00024-of-00027.safetensors
Waiting for inference service chaiml-reward-dpo-483b-30936-v5 to be ready
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00009-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00023-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00026-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00017-of-00027.safetensors
chaiml-reward-dpo-3a2b-96782-v4-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-3a2b-96782-v4/model-00020-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-63800-v25-uploader completed after 1663.73s with status: failed
Stopping job with name chaiml-kimid-v8b-kimid-63800-v25-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v8b-kimid-63800-v25-uploader
Waiting for job on chaiml-kimid-v8b-kimid-63800-v25-uploader to finish
Job chaiml-reward-dpo-3a2b-96782-v4-uploader completed after 1667.05s with status: succeeded
Stopping job with name chaiml-reward-dpo-3a2b-96782-v4-uploader
chaiml-kimid-v8b-kimid-63800-v25-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v8b-kimid-63800-v25-uploader: bash: no job control in this shell
Pipeline stage VLLMUploader completed in 1670.64s
chaiml-kimid-v8b-kimid-63800-v25-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
run pipeline stage %s
chaiml-kimid-v8b-kimid-63800-v25-uploader: __import__('pkg_resources').declare_namespace(__name__)
Running pipeline stage VLLMTemplater
chaiml-kimid-v8b-kimid-63800-v25-uploader: ╔═════════════════════════════════════════════════════════════════════╗
Pipeline stage VLLMTemplater completed in 0.70s
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
run pipeline stage %s
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ██████ ██████ █████ ████ ████ ║
Running pipeline stage VLLMDeployer
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
Creating inference service chaiml-reward-dpo-3a2b-96782-v4
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
Waiting for inference service chaiml-reward-dpo-3a2b-96782-v4 to be ready
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ https://mk1.ai ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ belonging to: ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v25-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8b-kimid-63800-v25-uploader: Downloaded to shared memory in 138.204s
chaiml-kimid-v8b-kimid-63800-v25-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-int4-mixed in 195.993s
chaiml-kimid-v8b-kimid-63800-v25-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-63800-v25-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-63800-v25-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/.gitattributes": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/README.md": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/added_tokens.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/chat_template.jinja": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/config.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/generation_config.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/merges.txt": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00001-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00002-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00003-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00004-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00005-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00006-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00007-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00008-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00009-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00010-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00011-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00012-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00013-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00014-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00015-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00016-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00017-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00018-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00019-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00020-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00021-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00022-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00023-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00024-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00026-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00027-of-00027.safetensors": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model.safetensors.index.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/quantization_config.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/special_tokens_map.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/tokenizer.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/tokenizer_config.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/vocab.json": object size matches
chaiml-kimid-v8b-kimid-63800-v25-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v25/model-00025-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-63800-v25-uploader completed after 281.91s with status: succeeded
Stopping job with name chaiml-kimid-v8b-kimid-63800-v25-uploader
Pipeline stage VLLMUploader completed in 1949.91s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.57s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8b-kimid-63800-v25
Waiting for inference service chaiml-kimid-v8b-kimid-63800-v25 to be ready
Inference service chaiml-reward-dpo-483b-30936-v5 ready after 457.3573350906372s
Pipeline stage VLLMDeployer completed in 459.64s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.951805353164673s
Received healthy response to inference request in 2.287142276763916s
Received healthy response to inference request in 2.7931125164031982s
Received healthy response to inference request in 2.6247992515563965s
Received healthy response to inference request in 2.7661116123199463s
Received healthy response to inference request in 2.4032363891601562s
Received healthy response to inference request in 2.6112329959869385s
Received healthy response to inference request in 2.2613718509674072s
Received healthy response to inference request in 2.6068115234375s
Received healthy response to inference request in 2.716684103012085s
Received healthy response to inference request in 2.6417741775512695s
Received healthy response to inference request in 3.281804084777832s
Received healthy response to inference request in 2.491016149520874s
Received healthy response to inference request in 2.5212860107421875s
Received healthy response to inference request in 2.1873700618743896s
Received healthy response to inference request in 2.4175381660461426s
Received healthy response to inference request in 2.360775947570801s
Received healthy response to inference request in 2.2840590476989746s
Received healthy response to inference request in 2.603428840637207s
Received healthy response to inference request in 2.274653434753418s
Received healthy response to inference request in 2.350966691970825s
Received healthy response to inference request in 2.324223518371582s
Received healthy response to inference request in 2.9296908378601074s
Received healthy response to inference request in 2.344580888748169s
Received healthy response to inference request in 2.484020709991455s
Received healthy response to inference request in 2.4747493267059326s
Received healthy response to inference request in 2.294299840927124s
Received healthy response to inference request in 2.5429608821868896s
Received healthy response to inference request in 2.2925901412963867s
Received healthy response to inference request in 2.557417869567871s
30 requests
0 failed requests
5th percentile: 2.267348563671112
10th percentile: 2.283118486404419
20th percentile: 2.2939579010009767
30th percentile: 2.3490509510040285
40th percentile: 2.411817455291748
50th percentile: 2.4875184297561646
60th percentile: 2.5487436771392824
70th percentile: 2.6081379652023315
80th percentile: 2.6567561626434326
90th percentile: 2.8067703485488895
95th percentile: 2.9418538212776184
99th percentile: 3.186104452610016
mean time: 2.5227171500523884
Pipeline stage StressChecker completed in 98.08s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.59s
Shutdown handler de-registered
chaiml-reward-dpo-483b-_30936_v5 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 3119.12s
Shutdown handler de-registered
chaiml-reward-dpo-483b-_30936_v5 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-reward-dpo-483b-_30936_v5
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-reward-dpo-483b-30936-v5 is running
Tearing down inference service chaiml-reward-dpo-483b-30936-v5
Service chaiml-reward-dpo-483b-30936-v5 has been torndown
Pipeline stage VLLMDeleter completed in 1.37s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-reward-dpo-483b-30936-v5/.gitattributes from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/added_tokens.json from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/config.json from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/merges.txt from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00001-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00002-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00003-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00004-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00005-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00006-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00007-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00008-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00009-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00010-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00011-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00012-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00013-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00014-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00015-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00016-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00017-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00018-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00019-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00020-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00021-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00022-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00023-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00024-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00025-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00026-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model-00027-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/quantization_config.json from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/tokenizer_config.json from bucket guanaco-vllm-models
Deleting key chaiml-reward-dpo-483b-30936-v5/vocab.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 13.52s
Shutdown handler de-registered
HTTP Request: %s %s "%s %d %s"
chaiml-reward-dpo-483b-_30936_v5 status is now torndown due to DeploymentManager action