Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Pipeline stage vllm_upload skipped, reason=amd cluster
Pipeline stage VLLMUploader completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Starting job with name chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd
Waiting for job on chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd to finish
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: {repo_id} is already quantized
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: Using quantization_mode: none
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: Downloading snapshot of ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-FP8...
2026-04-02T15:31:09.599930+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:32:09.704476+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: Downloaded in 121.282s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-02T15:33:09.796648+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-FP8 in 210.236s
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd:   if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/added_tokens.json
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/config.json
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/.gitattributes
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/chat_template.jinja
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/special_tokens_map.json
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/merges.txt
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/generation_config.json
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/recipe.yaml
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/tokenizer_config.json
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model.safetensors.index.json
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/vocab.json
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/tokenizer.json
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00048-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00048-of-00048.safetensors
2026-04-02T15:34:09.887113+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00023-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00023-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00027-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00027-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00018-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00018-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00025-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00025-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00019-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00019-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00033-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00033-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00021-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00021-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00006-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00006-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00015-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00015-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00008-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00008-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00029-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00029-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00024-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00024-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00020-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00020-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00039-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00039-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00003-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00003-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00007-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00007-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00041-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00041-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00028-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00028-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00016-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00016-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00034-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00034-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00038-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00038-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00031-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00031-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00047-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00047-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00045-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00045-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00040-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00040-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00042-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00042-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00044-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00044-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00014-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00014-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00026-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00026-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00010-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00010-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00037-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00037-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00009-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00009-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00011-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00011-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00022-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00022-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00001-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00001-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00035-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00035-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00002-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00002-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00043-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00043-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00032-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00032-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00004-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00004-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00030-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00030-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00036-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00036-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00012-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00012-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00005-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00005-of-00048.safetensors
chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd: cp /dev/shm/model_output/model-00046-of-00048.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd/model-00046-of-00048.safetensors
Job chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd completed after 289.98s with status: succeeded
Stopping job with name chaiml-kimid-v8b-kimidv-81429-v7-uploader-amd
Pipeline stage VLLMUploaderAMD completed in 290.51s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.28s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8b-kimidv-81429-v7
Waiting for inference service chaiml-kimid-v8b-kimidv-81429-v7 to be ready
2026-04-02T15:35:10.004662+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:36:10.121273+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:37:10.212732+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:38:10.297635+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:39:10.405617+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:40:10.537183+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:41:10.785116+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:42:10.932130+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:43:11.081244+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:44:11.182828+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:45:11.290888+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:46:11.390504+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:47:11.492313+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:48:11.647759+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
Retrying (%r) after connection broken by '%r': %s
2026-04-02T15:49:11.794989+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:50:11.916026+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:51:12.015449+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:52:12.128038+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:53:12.247995+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:54:12.726041+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
Retrying (%r) after connection broken by '%r': %s
Failed to get response for submission chaiml-qwen-bobo-dpo-ju_56781_v7: ('http://chaiml-qwen-bobo-dpo-ju-56781-v7-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
2026-04-02T15:55:12.903554+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:56:13.023031+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:57:13.204708+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:58:13.412866+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T15:59:13.519865+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:00:13.620726+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:01:13.862553+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:02:13.971484+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:03:14.123238+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:04:14.239023+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:05:14.449146+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:06:14.556448+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:07:14.681136+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:08:14.831870+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:09:14.980460+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
Failed to get response for submission chaiml-gspo-glm47-combi_10268_v3: ('http://chaiml-gspo-glm47-combi-10268-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'request timeout')
2026-04-02T16:10:15.100596+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:11:15.206628+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:12:15.304705+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:13:15.405711+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:14:15.723167+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
2026-04-02T16:15:15.824724+00:00 monitor updated for chaiml-kimid-v8b-kimidv_81429_v7
Tearing down inference service chaiml-kimid-v8b-kimidv-81429-v7
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-kimid-v8b-kimidv-81429-v7.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4800\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'40m\'}, \'creationTimestamp\': \'2026-04-02T15:35:04Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'prometheus.k.chaiverse.com\': \'true\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:prometheus.k.chaiverse.com\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:ion\': {}}, \'f:podAffinity\': {\'.\': {}, \'f:preferredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-04-02T15:35:04Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-04-02T15:35:04Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'f:autoscaling.knative.dev/initial-scale\': {}}}}, \'manager\': \'kubectl-edit\', \'operation\': \'Update\', \'time\': \'2026-04-02T15:38:40Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-04-02T15:38:41Z\'}], \'name\': \'chaiml-kimid-v8b-kimidv-81429-v7\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'489295993\', \'uid\': \'0b860026-2d44-4450-832d-f8a6f6d9f1aa\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'ion\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'amd.com/gpu.product-name\', \'operator\': \'In\', \'values\': [\'AMD_Instinct_MI325DLC_OAM\', \'AMD_Instinct_MI300X_OAM\']}, {\'key\': \'dcm.amd.com/gpu-config-profile\', \'operator\': \'In\', \'values\': [\'spx-profile\']}]}]}}, \'podAffinity\': {\'preferredDuringSchedulingIgnoredDuringExecution\': [{\'podAffinityTerm\': {\'labelSelector\': {\'matchLabels\': {\'serving.kserve.io/inferenceservice\': \'chaiml-kimid-v8b-kimidv-81429-v7\'}}, \'topologyKey\': \'kubernetes.io/hostname\'}, \'weight\': 100}]}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-kimid-v8b-kimidv-81429-v7/amd\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'4\', \'--gpu-memory-utilization\', \'0.9\', \'--max-model-len\', \'32768\', \'--max-num-batched-tokens\', \'65536\', \'--max-num-seqs\', \'512\', \'--kv-cache-dtype\', \'fp8\', \'--quantization\', \'compressed-tensors\', \'--enable-expert-parallel\', \'--trust-remote-code\', \'--attention-backend\', \'ROCM_AITER_FA\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-FP8\', \'--model-loader-extra-config\', \'{"distributed": true, "concurrency": 2}\', \'--compilation-config\', \'{"cudagraph_mode": "PIECEWISE", "max_cudagraph_capture_size": 512}\', \'--distributed-executor-backend\', \'mp\', \'--enable-prefix-caching\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'4\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'1\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'CWZAGMHZXKZRFGJK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'cwoAeWzp46q4O0sTNXOEuZ1MvZzKEFlS9DtEhnTldKp\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://cwobject.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}, {\'name\': \'RUNAI_STREAMER_S3_REQUEST_TIMEOUT_MS\', \'value\': \'30000\'}, {\'name\': \'VLLM_ROCM_USE_AITER\', \'value\': \'1\'}, {\'name\': \'VLLM_ROCM_USE_AITER_MOE\', \'value\': \'1\'}, {\'name\': \'AITER_ONLINE_TUNE\', \'value\': \'1\'}, {\'name\': \'VLLM_ROCM_QUICK_REDUCE_QUANTIZATION\', \'value\': \'INT4\'}, {\'name\': \'VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT\', \'value\': \'1\'}], \'image\': \'gcr.io/chai-959f8/vllm_amd:v0.17.1\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'amd.com/gpu\': \'4\', \'cpu\': \'8\', \'memory\': \'268Gi\'}, \'requests\': {\'amd.com/gpu\': \'4\', \'cpu\': \'8\', \'memory\': \'268Gi\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 10, \'minReplicas\': 0, \'priorityClassName\': \'chaiverse\', \'timeout\': 20, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'268Gi\'}, \'name\': \'shared-memory-cache\'}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-kimid-v8b-kimidv-81429-v7-predictor-00002\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-04-02T15:38:41Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-04-02T15:38:41Z\', \'message\': \'Revision "chaiml-kimid-v8b-kimidv-81429-v7-predictor-00002" failed with message: 0/81 nodes are available: 1 node(s) had untolerated taint {glm5-test: true}, 3 node(s) didn\\\'t match Pod\\\'s node affinity/selector, 6 node(s) were unschedulable, 71 Insufficient amd.com/gpu..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-04-02T15:38:41Z\', \'message\': \'Configuration "chaiml-kimid-v8b-kimidv-81429-v7-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-04-02T15:38:41Z\', \'message\': \'Configuration "chaiml-kimid-v8b-kimidv-81429-v7-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-04-02T15:38:41Z\', \'message\': \'Configuration "chaiml-kimid-v8b-kimidv-81429-v7-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-04-02T15:38:41Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 2}}')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-kimid-v8b-kimidv-81429-v7 is running
Skipping teardown as no inference service was found
Pipeline stage VLLMDeleter completed in 0.51s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Skipping deletion as no model was successfully uploaded
Pipeline stage VLLMModelDeleter completed in 0.21s
Shutdown handler de-registered
chaiml-kimid-v8b-kimidv_81429_v7 status is now failed due to DeploymentManager action
chaiml-kimid-v8b-kimidv_81429_v7 status is now torndown due to DeploymentManager action
chaiml-kimid-v8b-kimidv_81429_v7 status is now deployed due to DeploymentManager action
chaiml-kimid-v8b-kimidv_81429_v7 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v8b-kimidv_81429_v7 status is now torndown due to DeploymentManager action