Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-83165-v17-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v17-uploader to finish
chaiml-qwen-235b-95p-5f-92674-v3-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-qwen-235b-95p-5f-92674-v3-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: bash: no job control in this shell
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ https://mk1.ai ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ belonging to: ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ https://mk1.ai ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ belonging to: ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-qwen-235b-95p-5f-92674-v3-uploader: Downloaded to shared memory in 147.854s
chaiml-kimid-v9-opusdv-83165-v17-uploader: Downloaded to shared memory in 158.502s
chaiml-qwen-235b-95p-5f-92674-v3-uploader: Processed model ChaiML/qwen_235B_95p_5f_reward_dpo_10k_samples_opusdv1-int4-mixed in 200.603s
chaiml-qwen-235b-95p-5f-92674-v3-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-235b-95p-5f-92674-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-235b-95p-5f-92674-v3-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/added_tokens.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/chat_template.jinja
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/.gitattributes
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/special_tokens_map.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/tokenizer_config.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/generation_config.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/merges.txt
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/quantization_config.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/config.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/vocab.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model.safetensors.index.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/tokenizer.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-int4-mixed in 209.817s
chaiml-kimid-v9-opusdv-83165-v17-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-83165-v17-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-83165-v17-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/.gitattributes
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/added_tokens.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/chat_template.jinja
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/README.md
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/tokenizer_config.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/generation_config.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/special_tokens_map.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/quantization_config.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/config.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/tokenizer.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/merges.txt
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/vocab.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v9-opusdv-83165-v17-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00016-of-00027.safetensors?partNumber=18&uploadId=2~DmM0zErtkMt4Aca_Qr0bke0RFJzHK0Y": write tcp 10.0.46.246:56160->216.153.53.63:443: write: connection reset by peer
chaiml-kimid-v9-opusdv-83165-v17-uploader: ERROR "cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00016-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~DmM0zErtkMt4Aca_Qr0bke0RFJzHK0Y caused by: SignatureDoesNotMatch: status code: 403, request id: tx00000a460ae867d43fc2d-00697a46d7-14a19326cd-default, host id:
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00027-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00027-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00010-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00015-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00025-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00007-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00021-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00013-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00022-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00009-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00016-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00004-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00020-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00015-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00012-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00017-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00008-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00019-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00011-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00003-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00010-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00001-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00004-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00025-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-83165-v17-uploader completed after 773.46s with status: failed
Stopping job with name chaiml-kimid-v9-opusdv-83165-v17-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v9-opusdv-83165-v17-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v17-uploader to finish
chaiml-kimid-v9-opusdv-83165-v17-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v9-opusdv-83165-v17-uploader: bash: no job control in this shell
chaiml-kimid-v9-opusdv-83165-v17-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v9-opusdv-83165-v17-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v9-opusdv-83165-v17-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
Job chaiml-qwen-235b-95p-5f-92674-v3-uploader completed after 781.19s with status: succeeded
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
Stopping job with name chaiml-qwen-235b-95p-5f-92674-v3-uploader
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
Pipeline stage VLLMUploader completed in 783.26s
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
run pipeline stage %s
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ █████ █████ █████ ░░████ █████ ║
Running pipeline stage VLLMTemplater
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
Pipeline stage VLLMTemplater completed in 0.64s
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Version: 0.30.6+torch280 ║
run pipeline stage %s
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Features: FLYWHEEL, CUDA ║
Running pipeline stage VLLMDeployer
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
Creating inference service chaiml-qwen-235b-95p-5f-92674-v3
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ https://mk1.ai ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
Waiting for inference service chaiml-qwen-235b-95p-5f-92674-v3 to be ready
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ belonging to: ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v9-opusdv-83165-v17-uploader: Downloaded to shared memory in 151.290s
chaiml-kimid-v9-opusdv-83165-v17-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-int4-mixed in 214.294s
chaiml-kimid-v9-opusdv-83165-v17-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-83165-v17-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-83165-v17-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/.gitattributes": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/README.md": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/added_tokens.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/chat_template.jinja": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/config.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/generation_config.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/merges.txt": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00001-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00002-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00003-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00004-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00005-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00006-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00007-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00008-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00009-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00010-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00011-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00012-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00013-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00014-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00015-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00017-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00018-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00019-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00020-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00021-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00022-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00023-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00024-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00025-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00026-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00027-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model.safetensors.index.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/quantization_config.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/special_tokens_map.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/tokenizer.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/tokenizer_config.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/vocab.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00016-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-83165-v17-uploader completed after 263.3s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-83165-v17-uploader
Pipeline stage VLLMUploader completed in 1040.86s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.55s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-83165-v17
Waiting for inference service chaiml-kimid-v9-opusdv-83165-v17 to be ready
Inference service chaiml-qwen-235b-95p-5f-92674-v3 ready after 437.5312707424164s
Pipeline stage VLLMDeployer completed in 439.37s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.9888055324554443s
Received healthy response to inference request in 2.9072659015655518s
Received healthy response to inference request in 2.1958885192871094s
Received healthy response to inference request in 2.5546646118164062s
Received healthy response to inference request in 2.4633312225341797s
Received healthy response to inference request in 2.2723567485809326s
Received healthy response to inference request in 2.280489444732666s
Received healthy response to inference request in 2.222970962524414s
Received healthy response to inference request in 2.385891914367676s
Received healthy response to inference request in 2.530067205429077s
Received healthy response to inference request in 2.144638776779175s
Received healthy response to inference request in 2.317072868347168s
Received healthy response to inference request in 2.339120864868164s
Received healthy response to inference request in 2.862046957015991s
Received healthy response to inference request in 2.1863958835601807s
Received healthy response to inference request in 2.3233320713043213s
Received healthy response to inference request in 2.3718531131744385s
Received healthy response to inference request in 2.2574870586395264s
Received healthy response to inference request in 2.201815366744995s
Received healthy response to inference request in 2.8437156677246094s
Received healthy response to inference request in 2.494262456893921s
Received healthy response to inference request in 2.2720706462860107s
Received healthy response to inference request in 2.2844760417938232s
Received healthy response to inference request in 2.2059648036956787s
Received healthy response to inference request in 2.626243829727173s
Received healthy response to inference request in 2.4604499340057373s
Received healthy response to inference request in 2.7456774711608887s
Received healthy response to inference request in 2.344508171081543s
Received healthy response to inference request in 3.046015501022339s
Received healthy response to inference request in 2.5127720832824707s
30 requests
0 failed requests
5th percentile: 2.1906675696372986
10th percentile: 2.2012226819992065
20th percentile: 2.250583839416504
30th percentile: 2.278049635887146
40th percentile: 2.32082839012146
50th percentile: 2.3581806421279907
60th percentile: 2.4616024494171143
70th percentile: 2.5179606199264524
80th percentile: 2.6501305580139163
90th percentile: 2.8665688514709475
95th percentile: 2.9521126985549926
99th percentile: 3.0294246101379394
mean time: 2.454721721013387
Pipeline stage StressChecker completed in 90.53s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.89s
Shutdown handler de-registered
chaiml-qwen-235b-95p-5f_92674_v3 status is now deployed due to DeploymentManager action
Inference service chaiml-kimid-v9-opusdv-83165-v17 ready after 446.4351387023926s
Pipeline stage VLLMDeployer completed in 447.83s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.7682437896728516s
Received healthy response to inference request in 2.714886426925659s
Received healthy response to inference request in 3.084115982055664s
Received healthy response to inference request in 2.1617493629455566s
Received healthy response to inference request in 2.2621569633483887s
Received healthy response to inference request in 2.2571394443511963s
Received healthy response to inference request in 2.501967191696167s
Received healthy response to inference request in 2.3694989681243896s
Received healthy response to inference request in 2.330413818359375s
Received healthy response to inference request in 2.279550075531006s
Received healthy response to inference request in 2.315011501312256s
Received healthy response to inference request in 2.1874442100524902s
Received healthy response to inference request in 2.216820478439331s
Received healthy response to inference request in 2.7934608459472656s
Received healthy response to inference request in 2.2784581184387207s
Received healthy response to inference request in 2.9735288619995117s
Received healthy response to inference request in 2.143725633621216s
Received healthy response to inference request in 2.33367657661438s
Received healthy response to inference request in 2.2369723320007324s
Received healthy response to inference request in 2.172360420227051s
Received healthy response to inference request in 2.4386324882507324s
Received healthy response to inference request in 2.623349905014038s
Received healthy response to inference request in 2.3081600666046143s
Received healthy response to inference request in 2.2345659732818604s
Received healthy response to inference request in 2.5073201656341553s
Received healthy response to inference request in 2.1788437366485596s
Received healthy response to inference request in 2.3869102001190186s
Received healthy response to inference request in 2.2355668544769287s
Received healthy response to inference request in 2.597720146179199s
Received healthy response to inference request in 2.569599151611328s
30 requests
0 failed requests
5th percentile: 2.166524338722229
10th percentile: 2.1781954050064085
20th percentile: 2.2310168743133545
30th percentile: 2.251089310646057
40th percentile: 2.279113292694092
50th percentile: 2.3227126598358154
60th percentile: 2.376463460922241
70th percentile: 2.5035730838775634
80th percentile: 2.602846097946167
90th percentile: 2.770765495300293
95th percentile: 2.8924982547760005
99th percentile: 3.05204571723938
mean time: 2.415394989649455
Pipeline stage StressChecker completed in 81.20s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.37s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_83165_v17 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 2868.03s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_83165_v17 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v9-opusdv_83165_v17 status is now torndown due to DeploymentManager action