Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v11-kimidv-71337-v4-uploader
Waiting for job on chaiml-kimid-v11-kimidv-71337-v4-uploader to finish
chaiml-kimid-v11-kimidv-71337-v4-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v11-kimidv-71337-v4-uploader: bash: no job control in this shell
chaiml-kimid-v11-kimidv-71337-v4-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v11-kimidv-71337-v4-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v11-kimidv-71337-v4-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ https://mk1.ai ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ belonging to: ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v11-kimidv-71337-v4-uploader: Downloaded to shared memory in 124.864s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-71337-v4-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v11-kimidv-71337-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v11-kimidv-71337-v4-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/.gitattributes
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/chat_template.jinja
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/generation_config.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/tokenizer_config.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/added_tokens.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/config.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/quantization_config.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/special_tokens_map.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/vocab.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/tokenizer.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/merges.txt
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model.safetensors.index.json
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v11-kimidv-71337-v4-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00011-of-00027.safetensors?partNumber=2&uploadId=2~XfRds8GUqpF-eAVoeYD-K7_p5l_4csE": write tcp 10.0.1.214:47034->216.153.53.63:443: write: connection reset by peer
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v11-kimidv-71337-v4-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00014-of-00027.safetensors?partNumber=1&uploadId=2~UTUNi-Wr_55yb3FGy_RdmD0kpu6EEDI": write tcp 10.0.1.214:47046->216.153.53.63:443: write: connection reset by peer
chaiml-kimid-v11-kimidv-71337-v4-uploader: ERROR "cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00014-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~UTUNi-Wr_55yb3FGy_RdmD0kpu6EEDI caused by: SignatureDoesNotMatch: status code: 403, request id: tx000004430a97d9ccd8351-006974d437-14af544255-default, host id:
chaiml-kimid-v11-kimidv-71337-v4-uploader: ERROR "cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00011-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~XfRds8GUqpF-eAVoeYD-K7_p5l_4csE caused by: SignatureDoesNotMatch: status code: 403, request id: tx0000083d5d2ec06c00efd-006974d434-14a24de22e-default, host id:
Failed to get response for submission blend_litaf_2026-01-22: HTTPConnectionPool(host='guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00021-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00004-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00020-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00001-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00002-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00015-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00003-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00009-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00005-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00008-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00023-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00022-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00025-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00012-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00006-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00007-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00018-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00019-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00017-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00010-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00026-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00013-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00024-of-00027.safetensors
Job chaiml-kimid-v11-kimidv-71337-v4-uploader completed after 695.31s with status: failed
Stopping job with name chaiml-kimid-v11-kimidv-71337-v4-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v11-kimidv-71337-v4-uploader
Waiting for job on chaiml-kimid-v11-kimidv-71337-v4-uploader to finish
chaiml-kimid-v11-kimidv-71337-v4-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v11-kimidv-71337-v4-uploader: bash: no job control in this shell
chaiml-kimid-v11-kimidv-71337-v4-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v11-kimidv-71337-v4-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v11-kimidv-71337-v4-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ https://mk1.ai ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ belonging to: ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ║ ║
chaiml-kimid-v11-kimidv-71337-v4-uploader: ╚═════════════════════════════════════════════════════════════════════╝
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-71337-v4-uploader: Downloaded to shared memory in 130.324s
chaiml-kimid-v11-kimidv-71337-v4-uploader: Processed model ChaiML/kimid-v11-kimidv4ann-lr1e5ep2r64g8b01-int4-mixed in 191.804s
chaiml-kimid-v11-kimidv-71337-v4-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v11-kimidv-71337-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v11-kimidv-71337-v4-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/.gitattributes": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/added_tokens.json": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/chat_template.jinja": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/config.json": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/generation_config.json": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/merges.txt": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00001-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00002-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00003-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00004-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00005-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00006-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00007-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00008-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00009-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00010-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00012-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00013-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00015-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00016-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00017-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00018-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00019-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00020-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00021-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00022-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00023-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00024-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00025-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00026-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00027-of-00027.safetensors": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model.safetensors.index.json": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/quantization_config.json": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/special_tokens_map.json": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/tokenizer.json": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/tokenizer_config.json": object size matches
chaiml-kimid-v11-kimidv-71337-v4-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/vocab.json": object size matches
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00011-of-00027.safetensors
chaiml-kimid-v11-kimidv-71337-v4-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-71337-v4/model-00014-of-00027.safetensors
Job chaiml-kimid-v11-kimidv-71337-v4-uploader completed after 299.71s with status: succeeded
Stopping job with name chaiml-kimid-v11-kimidv-71337-v4-uploader
Pipeline stage VLLMUploader completed in 996.66s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.97s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v11-kimidv-71337-v4
Waiting for inference service chaiml-kimid-v11-kimidv-71337-v4 to be ready
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-kimid-v11-kimidv-71337-v4 ready after 492.5936162471771s
Pipeline stage VLLMDeployer completed in 493.32s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.294807195663452s
Received healthy response to inference request in 1.777151107788086s
Received healthy response to inference request in 2.2847745418548584s
Received healthy response to inference request in 1.7540409564971924s
Received healthy response to inference request in 1.7698304653167725s
Received healthy response to inference request in 1.7732930183410645s
Received healthy response to inference request in 1.9700355529785156s
Received healthy response to inference request in 1.8707735538482666s
Received healthy response to inference request in 2.3639988899230957s
Received healthy response to inference request in 1.7900326251983643s
Received healthy response to inference request in 2.665600299835205s
Received healthy response to inference request in 2.1936275959014893s
Received healthy response to inference request in 2.2406344413757324s
Received healthy response to inference request in 1.793607234954834s
Received healthy response to inference request in 2.122532606124878s
Received healthy response to inference request in 1.813302993774414s
Received healthy response to inference request in 1.8992269039154053s
Received healthy response to inference request in 1.789912223815918s
Received healthy response to inference request in 4.076100587844849s
Received healthy response to inference request in 3.2539830207824707s
Received healthy response to inference request in 1.8249168395996094s
Received healthy response to inference request in 2.137998580932617s
Received healthy response to inference request in 1.8182644844055176s
Received healthy response to inference request in 1.757643699645996s
Received healthy response to inference request in 1.7726986408233643s
Received healthy response to inference request in 2.059844970703125s
Received healthy response to inference request in 1.860645055770874s
Received healthy response to inference request in 1.818514347076416s
Received healthy response to inference request in 1.7718307971954346s
Received healthy response to inference request in 1.8373830318450928s
30 requests
0 failed requests
5th percentile: 1.7631277441978455
10th percentile: 1.7716307640075684
20th percentile: 1.7763794898986816
30th percentile: 1.792534852027893
40th percentile: 1.8184144020080566
50th percentile: 1.8490140438079834
60th percentile: 1.9275503635406492
70th percentile: 2.1271723985671995
80th percentile: 2.2494624614715577
90th percentile: 2.3941590309143073
95th percentile: 2.9892107963561996
99th percentile: 3.8376864933967596
mean time: 2.071900208791097
Pipeline stage StressChecker completed in 65.99s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-kimid-v11-kimidv_71337_v4 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 4781.92s
Shutdown handler de-registered
chaiml-kimid-v11-kimidv_71337_v4 status is now torndown due to DeploymentManager action