Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v8a-kimid-94399-v13-uploader
Waiting for job on chaiml-kimid-v8a-kimid-94399-v13-uploader to finish
chaiml-kimid-v8a-kimid-94399-v13-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v8a-kimid-94399-v13-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v8a-kimid-94399-v13-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ https://mk1.ai ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ belonging to: ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8a-kimid-94399-v13-uploader: Downloaded to shared memory in 95.408s
chaiml-kimid-v8a-kimid-94399-v13-uploader: Processed model ChaiML/kimid-v8a-kimidv6-lr5e6ep2r64g2b01-int4-mixed in 164.730s
chaiml-kimid-v8a-kimid-94399-v13-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/merges.txt
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/vocab.json
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model.safetensors.index.json
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/tokenizer.json
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00027-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v8a-kimid-94399-v13-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00010-of-00027.safetensors?partNumber=67&uploadId=2~9_IYG6HBTEbNwYM1b6b5R_ENAVa-14b": write tcp 10.0.1.97:43222->216.153.53.63:443: write: connection reset by peer
chaiml-kimid-v8a-kimid-94399-v13-uploader: ERROR "cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00010-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~9_IYG6HBTEbNwYM1b6b5R_ENAVa-14b caused by: SignatureDoesNotMatch: status code: 403, request id: tx000003bd8f68352dd4d32-006961a4df-14b31daae8-default, host id:
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00005-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00020-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00013-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00025-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00023-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00011-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00022-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00007-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00012-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00015-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00001-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00008-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00017-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00026-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00016-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00014-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00004-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00006-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00009-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00003-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00019-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00021-of-00027.safetensors
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00002-of-00027.safetensors
Job chaiml-kimid-v8a-kimid-94399-v13-uploader completed after 361.66s with status: failed
Stopping job with name chaiml-kimid-v8a-kimid-94399-v13-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v8a-kimid-94399-v13-uploader
Waiting for job on chaiml-kimid-v8a-kimid-94399-v13-uploader to finish
chaiml-kimid-v8a-kimid-94399-v13-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v8a-kimid-94399-v13-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v8a-kimid-94399-v13-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ https://mk1.ai ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ belonging to: ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ║ ║
chaiml-kimid-v8a-kimid-94399-v13-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8a-kimid-94399-v13-uploader: Downloaded to shared memory in 94.736s
chaiml-kimid-v8a-kimid-94399-v13-uploader: Processed model ChaiML/kimid-v8a-kimidv6-lr5e6ep2r64g2b01-int4-mixed in 164.509s
chaiml-kimid-v8a-kimid-94399-v13-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8a-kimid-94399-v13-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8a-kimid-94399-v13-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/.gitignore s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/.gitignore": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/.gitattributes.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/.gitattributes.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/README.md.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/README.md.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/README.md.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/README.md.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/added_tokens.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/added_tokens.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/added_tokens.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/added_tokens.json.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/chat_template.jinja.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/chat_template.jinja.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/chat_template.jinja.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/chat_template.jinja.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/config.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/config.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/config.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/config.json.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/generation_config.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/generation_config.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/generation_config.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/generation_config.json.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/merges.txt.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/merges.txt.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/merges.txt.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/merges.txt.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00001-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00001-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00002-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00002-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00002-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00002-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00003-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00003-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00003-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00003-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00004-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00004-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00004-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00004-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00005-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00005-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00006-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00006-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00006-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00006-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00007-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00007-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00007-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00007-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00008-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00008-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00008-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00008-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00009-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00009-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00010-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00010-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00010-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00010-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00011-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00011-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00011-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00011-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00012-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00012-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00013-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00013-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00014-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00014-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00014-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00014-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00015-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00015-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00016-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00016-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00016-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00016-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00017-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00017-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00017-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00017-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00018-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00018-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00018-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00018-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00019-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00019-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00019-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00019-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00020-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00020-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00020-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00020-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00021-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00021-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00021-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00021-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00022-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00022-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00023-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00023-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00023-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00023-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00024-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00024-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00025-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00025-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00025-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00025-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00026-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00026-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00027-of-00027.safetensors.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00027-of-00027.safetensors.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model-00027-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00027-of-00027.safetensors.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model.safetensors.index.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model.safetensors.index.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/model.safetensors.index.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model.safetensors.index.json.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/quantization_config.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/quantization_config.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/quantization_config.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/quantization_config.json.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/special_tokens_map.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/special_tokens_map.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/special_tokens_map.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/special_tokens_map.json.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/tokenizer.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/tokenizer.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/tokenizer_config.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/tokenizer_config.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/vocab.json.lock s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/vocab.json.lock": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.cache/huggingface/download/vocab.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/vocab.json.metadata": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.gitattributes": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/README.md": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/added_tokens.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/chat_template.jinja": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/config.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/generation_config.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/merges.txt": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00001-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00002-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00003-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00004-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00005-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00006-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00007-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00008-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00009-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00011-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00012-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00013-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00014-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00015-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00016-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00017-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00018-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00019-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00020-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00021-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00022-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00023-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00024-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00025-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00026-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00027-of-00027.safetensors": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model.safetensors.index.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/quantization_config.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/special_tokens_map.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/tokenizer.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/tokenizer_config.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/vocab.json": object size matches
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/tokenizer.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/tokenizer.json.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00001-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00001-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/.gitattributes.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/.gitattributes.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00009-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00009-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00012-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00012-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00013-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00013-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00005-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00005-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00026-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00026-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00024-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00024-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/tokenizer_config.json.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/tokenizer_config.json.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00015-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00015-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/.cache/huggingface/download/model-00022-of-00027.safetensors.metadata s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/.cache/huggingface/download/model-00022-of-00027.safetensors.metadata
chaiml-kimid-v8a-kimid-94399-v13-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8a-kimid-94399-v13/model-00010-of-00027.safetensors
Job chaiml-kimid-v8a-kimid-94399-v13-uploader completed after 219.75s with status: succeeded
Stopping job with name chaiml-kimid-v8a-kimid-94399-v13-uploader
Pipeline stage VLLMUploader completed in 583.70s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.31s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8a-kimid-94399-v13
Waiting for inference service chaiml-kimid-v8a-kimid-94399-v13 to be ready
Inference service chaiml-kimid-v8a-kimid-94399-v13 ready after 457.02390480041504s
Pipeline stage VLLMDeployer completed in 458.89s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1362321376800537s
Received healthy response to inference request in 2.038236379623413s
Received healthy response to inference request in 1.8536529541015625s
Received healthy response to inference request in 1.943039894104004s
Received healthy response to inference request in 1.672992467880249s
Received healthy response to inference request in 1.6626083850860596s
Received healthy response to inference request in 1.8616740703582764s
Received healthy response to inference request in 1.7608978748321533s
Received healthy response to inference request in 1.7825467586517334s
Received healthy response to inference request in 2.154444456100464s
Received healthy response to inference request in 1.6365845203399658s
Received healthy response to inference request in 1.8662292957305908s
Received healthy response to inference request in 1.8698196411132812s
Received healthy response to inference request in 1.7291533946990967s
Received healthy response to inference request in 1.9533448219299316s
Received healthy response to inference request in 1.6906287670135498s
Received healthy response to inference request in 1.9444475173950195s
Received healthy response to inference request in 1.684072732925415s
Received healthy response to inference request in 1.7294378280639648s
Received healthy response to inference request in 1.7172331809997559s
Received healthy response to inference request in 1.8479115962982178s
Received healthy response to inference request in 2.091836929321289s
Received healthy response to inference request in 2.033586263656616s
Received healthy response to inference request in 1.7652676105499268s
Received healthy response to inference request in 1.8790159225463867s
Received healthy response to inference request in 1.724755048751831s
Received healthy response to inference request in 2.099606990814209s
Received healthy response to inference request in 1.8268609046936035s
Received healthy response to inference request in 1.6704058647155762s
Received healthy response to inference request in 1.6275372505187988s
30 requests
0 failed requests
5th percentile: 1.648295259475708
10th percentile: 1.6696261167526245
20th percentile: 1.689317560195923
30th percentile: 1.727833890914917
40th percentile: 1.7635197162628173
50th percentile: 1.8373862504959106
60th percentile: 1.8634961605072022
70th percentile: 1.8982231140136716
80th percentile: 1.9693931102752689
90th percentile: 2.092613935470581
95th percentile: 2.1197508215904235
99th percentile: 2.149162883758545
mean time: 1.8418020486831665
Pipeline stage StressChecker completed in 61.36s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.56s
Shutdown handler de-registered
chaiml-kimid-v8a-kimid_94399_v13 status is now deployed due to DeploymentManager action
chaiml-kimid-v8a-kimid_94399_v13 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v8a-kimid_94399_v13 status is now torndown due to DeploymentManager action