Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v8b-kimid-77693-v17-uploader
Waiting for job on chaiml-kimid-v8b-kimid-77693-v17-uploader to finish
chaiml-kimid-v8b-kimid-77693-v17-uploader: Using quantization_mode: w4a16
chaiml-kimid-v8b-kimid-77693-v17-uploader: Checking if ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v8b-kimid-77693-v17-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v8b-kimid-77693-v17-uploader: Downloading snapshot of ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-W4A16...
2026-04-03T16:19:59.033023+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v17
chaiml-kimid-v8b-kimid-77693-v17-uploader: Downloaded in 55.045s
chaiml-kimid-v8b-kimid-77693-v17-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01 in 57.726s
chaiml-kimid-v8b-kimid-77693-v17-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-77693-v17-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v17-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v8b-kimid-77693-v17-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v8b-kimid-77693-v17-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v8b-kimid-77693-v17-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v17-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v17-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v17-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v17-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v17-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v8b-kimid-77693-v17-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v17-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v8b-kimid-77693-v17-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v8b-kimid-77693-v17-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v17-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v8b-kimid-77693-v17-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v8b-kimid-77693-v17-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-77693-v17-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/.gitattributes
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/special_tokens_map.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/quantization_config.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/config.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/vocab.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/generation_config.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/merges.txt
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/chat_template.jinja
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/tokenizer_config.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/added_tokens.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/tokenizer.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model.safetensors.index.json
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00027-of-00027.safetensors
2026-04-03T16:20:59.147019+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v17
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00017-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00021-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00003-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00008-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00014-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00005-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00015-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00023-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00009-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00010-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00001-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00022-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00020-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00006-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00025-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00024-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00012-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00002-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00004-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00011-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00026-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00018-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00019-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v17-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v17/default/model-00007-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-77693-v17-uploader completed after 145.26s with status: succeeded
Stopping job with name chaiml-kimid-v8b-kimid-77693-v17-uploader
Pipeline stage VLLMUploader completed in 145.78s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.53s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8b-kimid-77693-v17
Waiting for inference service chaiml-kimid-v8b-kimid-77693-v17 to be ready
2026-04-03T16:21:59.234411+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v17
2026-04-03T16:22:59.332895+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v17
2026-04-03T16:23:59.511441+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v17
Inference service chaiml-kimid-v8b-kimid-77693-v17 ready after 201.25241708755493s
Pipeline stage VLLMDeployer completed in 201.81s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.14006233215332s
Received healthy response to inference request in 3.4788455963134766s
Received healthy response to inference request in 1.4759759902954102s
2026-04-03T16:24:59.600717+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v17
Received healthy response to inference request in 1.4476070404052734s
Received healthy response to inference request in 3.5119566917419434s
Received healthy response to inference request in 1.5144517421722412s
Received healthy response to inference request in 1.5147478580474854s
Received healthy response to inference request in 1.6140563488006592s
Received healthy response to inference request in 4.1121439933776855s
Received healthy response to inference request in 1.54156494140625s
Received healthy response to inference request in 1.4725384712219238s
Received healthy response to inference request in 1.4719493389129639s
Received healthy response to inference request in 1.4791886806488037s
Received healthy response to inference request in 1.462120532989502s
Received healthy response to inference request in 1.7884869575500488s
Received healthy response to inference request in 1.5355327129364014s
Received healthy response to inference request in 1.5431921482086182s
Received healthy response to inference request in 1.8013205528259277s
Received healthy response to inference request in 1.6547045707702637s
Received healthy response to inference request in 1.4775187969207764s
Received healthy response to inference request in 1.4831111431121826s
Received healthy response to inference request in 1.5281596183776855s
Received healthy response to inference request in 1.5205891132354736s
Received healthy response to inference request in 1.4872050285339355s
Received healthy response to inference request in 1.5995848178863525s
Received healthy response to inference request in 1.4944877624511719s
Received healthy response to inference request in 3.527223587036133s
Received healthy response to inference request in 1.5822949409484863s
Received healthy response to inference request in 1.8333563804626465s
2026-04-03T16:25:59.692101+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v17
Received healthy response to inference request in 2.243323564529419s
30 requests
0 failed requests
5th percentile: 1.4665434956550598
10th percentile: 1.472479557991028
20th percentile: 1.4788547039031983
30th percentile: 1.492302942276001
40th percentile: 1.5182526111602783
50th percentile: 1.5385488271713257
60th percentile: 1.5892108917236327
70th percentile: 1.6948392868041988
80th percentile: 1.9153498172760022
90th percentile: 3.5134833812713624
95th percentile: 3.8489298105239853
99th percentile: 4.131966013908387
mean time: 1.9445767084757486
Pipeline stage StressChecker completed in 71.83s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.06s
Shutdown handler de-registered
chaiml-kimid-v8b-kimid_77693_v17 status is now deployed due to DeploymentManager action
chaiml-kimid-v8b-kimid_77693_v17 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v8b-kimid_77693_v17 status is now torndown due to DeploymentManager action