Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v11-kimidv-71337-v3-uploader
Waiting for job on chaiml-kimid-v11-kimidv-71337-v3-uploader to finish
chaiml-kimid-v11-kimidv-71337-v3-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v11-kimidv-71337-v3-uploader: bash: no job control in this shell
chaiml-kimid-v11-kimidv-71337-v3-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v11-kimidv-71337-v3-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v11-kimidv-71337-v3-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ https://mk1.ai ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ belonging to: ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ╚═════════════════════════════════════════════════════════════════════╝
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-kimid-v11-noname_44244_v1: HTTPConnectionPool(host='chaiml-kimid-v11-noname-44244-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-kimid-v11-kimidv-71337-v3-uploader: Downloaded to shared memory in 109.968s
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-71337-v3-uploader: Processed model ChaiML/kimid-v11-kimidv4ann-lr1e5ep2r64g8b01-int4-mixed in 169.687s
chaiml-kimid-v11-kimidv-71337-v3-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v11-kimidv-71337-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v11-kimidv-71337-v3-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/.gitattributes
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/tokenizer_config.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/generation_config.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/special_tokens_map.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/added_tokens.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/config.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/quantization_config.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/chat_template.jinja
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/vocab.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/merges.txt
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/tokenizer.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model.safetensors.index.json
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v11-kimidv-71337-v3-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00017-of-00027.safetensors?partNumber=16&uploadId=2~ZX6wRtexF8n1Z8MPk-n72ROqyOx1wh6": net/http: HTTP/1.x transport connection broken: read tcp 10.0.1.235:45218->216.153.53.63:443: read: connection reset by peer
chaiml-kimid-v11-kimidv-71337-v3-uploader: ERROR "cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00017-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~ZX6wRtexF8n1Z8MPk-n72ROqyOx1wh6 caused by: SignatureDoesNotMatch: status code: 403, request id: tx0000045104ba94d6ea749-006974bc30-14a24de22e-default, host id:
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00027-of-00027.safetensors
Failed to get response for submission chaiml-kimid-v11-noname_44244_v1: HTTPConnectionPool(host='chaiml-kimid-v11-noname-44244-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v11-kimidv-71337-v3-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00014-of-00027.safetensors?partNumber=43&uploadId=2~I-cfKCuoOasaX-XeZbbp9J9tVgndkBW": net/http: HTTP/1.x transport connection broken: read tcp 10.0.1.235:45636->216.153.53.63:443: read: connection reset by peer
chaiml-kimid-v11-kimidv-71337-v3-uploader: ERROR "cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00014-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~I-cfKCuoOasaX-XeZbbp9J9tVgndkBW caused by: SignatureDoesNotMatch: status code: 403, request id: tx00000a70c3f520df3fc3d-006974bc79-14a19326cd-default, host id:
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v11-kimidv-71337-v3-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00007-of-00027.safetensors?partNumber=54&uploadId=2~DRt3GlrtaLQItKEr2UH4vzuacguA8WJ": net/http: HTTP/1.x transport connection broken: read tcp 10.0.1.235:39662->216.153.53.63:443: read: connection reset by peer
chaiml-kimid-v11-kimidv-71337-v3-uploader: ERROR "cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00007-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~DRt3GlrtaLQItKEr2UH4vzuacguA8WJ caused by: SignatureDoesNotMatch: status code: 403, request id: tx00000aed137de9813c252-006974bc7e-14b31daae8-default, host id:
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00002-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00008-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00009-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00022-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00018-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00001-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00012-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00011-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00023-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00015-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00016-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00019-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00013-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00006-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00025-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00026-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00003-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00010-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00021-of-00027.safetensors
Job chaiml-kimid-v11-kimidv-71337-v3-uploader completed after 379.8s with status: failed
Stopping job with name chaiml-kimid-v11-kimidv-71337-v3-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v11-kimidv-71337-v3-uploader
Waiting for job on chaiml-kimid-v11-kimidv-71337-v3-uploader to finish
chaiml-kimid-v11-kimidv-71337-v3-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v11-kimidv-71337-v3-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v11-kimidv-71337-v3-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ https://mk1.ai ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ belonging to: ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v3-uploader: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-71337-v3-uploader: Downloaded to shared memory in 108.773s
chaiml-kimid-v11-kimidv-71337-v3-uploader: Processed model ChaiML/kimid-v11-kimidv4ann-lr1e5ep2r64g8b01-int4-mixed in 168.670s
chaiml-kimid-v11-kimidv-71337-v3-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v11-kimidv-71337-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v11-kimidv-71337-v3-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/.gitattributes": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/added_tokens.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/chat_template.jinja": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/config.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/generation_config.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/merges.txt": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00001-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00002-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00003-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00004-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00005-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00006-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00008-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00009-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00010-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00011-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00012-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00013-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00015-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00016-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00018-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00019-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00020-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00021-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00022-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00023-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00024-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00025-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00026-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00027-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model.safetensors.index.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/quantization_config.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/special_tokens_map.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/tokenizer.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/tokenizer_config.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/vocab.json": object size matches
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00017-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00007-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v3-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3/model-00014-of-00027.safetensors
Job chaiml-kimid-v11-kimidv-71337-v3-uploader completed after 259.41s with status: succeeded
Stopping job with name chaiml-kimid-v11-kimidv-71337-v3-uploader
Pipeline stage VLLMUploader completed in 640.32s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v11-kimidv-71337-v3
Waiting for inference service chaiml-kimid-v11-kimidv-71337-v3 to be ready
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-kimid-v11-noname_44244_v1: HTTPConnectionPool(host='chaiml-kimid-v11-noname-44244-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-kimid-v11-noname_44244_v1: HTTPConnectionPool(host='chaiml-kimid-v11-noname-44244-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-kimid-v11-noname_44244_v1: HTTPConnectionPool(host='chaiml-kimid-v11-noname-44244-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-kimid-v11-noname_44244_v1: HTTPConnectionPool(host='chaiml-kimid-v11-noname-44244-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-kimid-v11-noname_44244_v1: HTTPConnectionPool(host='chaiml-kimid-v11-noname-44244-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-bol-v1-opusdv1b-_13427_v1: HTTPConnectionPool(host='chaiml-bol-v1-opusdv1b-13427-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Tearing down inference service chaiml-kimid-v11-kimidv-71337-v3
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-kimid-v11-kimidv-71337-v3. The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2026-01-24T12:40:25Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-01-24T12:40:25Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-01-24T12:40:25Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-01-24T12:40:26Z\'}], \'name\': \'chaiml-kimid-v11-kimidv-71337-v3\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'381238857\', \'uid\': \'97b9095b-2805-448e-9539-09c418a4519e\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v3\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'2\', \'--max-model-len\', \'10240\', \'--max-num-batched-tokens\', \'10240\', \'--max-num-seqs\', \'64\', \'--gpu-memory-utilization\', \'0.92\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/kimid-v11-kimidv4ann-lr1e5ep2r64g8b01-int4-mixed\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'2\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'0\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://object.ord1.coreweave.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.13.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'4\', \'memory\': \'128Gi\', \'nvidia.com/gpu\': \'2\'}, \'requests\': {\'cpu\': \'4\', \'memory\': \'128Gi\', \'nvidia.com/gpu\': \'2\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 40, \'minReplicas\': 0, \'priorityClassName\': \'creator-studio\', \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'128Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-kimid-v11-kimidv-71337-v3-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-01-24T12:40:26Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-01-24T12:40:26Z\', \'message\': \'Revision "chaiml-kimid-v11-kimidv-71337-v3-predictor-00001" failed with message: 0/72 nodes are available: 5 Insufficient nvidia.com/gpu, 67 node(s) didn\\\'t match Pod\\\'s node affinity/selector. no new claims to deallocate, preemption: 0/72 nodes are available: 5 Insufficient nvidia.com/gpu, 67 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-01-24T12:40:26Z\', \'message\': \'Configuration "chaiml-kimid-v11-kimidv-71337-v3-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-01-24T12:40:26Z\', \'message\': \'Configuration "chaiml-kimid-v11-kimidv-71337-v3-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-01-24T12:40:26Z\', \'message\': \'Configuration "chaiml-kimid-v11-kimidv-71337-v3-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-01-24T12:40:26Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.22s
Shutdown handler de-registered
chaiml-kimid-v11-kimidv_71337_v3 status is now failed due to DeploymentManager action