Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v8b-kimid-77693-v16-uploader
Waiting for job on chaiml-kimid-v8b-kimid-77693-v16-uploader to finish
chaiml-kimid-v8b-kimid-77693-v16-uploader: Using quantization_mode: w4a16
chaiml-kimid-v8b-kimid-77693-v16-uploader: Checking if ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v8b-kimid-77693-v16-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v8b-kimid-77693-v16-uploader: Downloading snapshot of ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-W4A16...
2026-04-03T07:09:07.587479+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v16
chaiml-kimid-v8b-kimid-77693-v16-uploader: Downloaded in 67.575s
chaiml-kimid-v8b-kimid-77693-v16-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01 in 70.251s
chaiml-kimid-v8b-kimid-77693-v16-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-77693-v16-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v16-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v8b-kimid-77693-v16-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v8b-kimid-77693-v16-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v8b-kimid-77693-v16-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v16-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v16-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v16-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v16-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v16-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v8b-kimid-77693-v16-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v16-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v8b-kimid-77693-v16-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v8b-kimid-77693-v16-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v16-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v8b-kimid-77693-v16-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v8b-kimid-77693-v16-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-77693-v16-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/generation_config.json
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/added_tokens.json
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/chat_template.jinja
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/special_tokens_map.json
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/.gitattributes
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/merges.txt
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/quantization_config.json
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/config.json
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/tokenizer_config.json
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model.safetensors.index.json
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/vocab.json
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/tokenizer.json
2026-04-03T07:10:07.670427+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v16
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00027-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00026-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00019-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00005-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00015-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00023-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00007-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00018-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00009-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00011-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00006-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00003-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00024-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00002-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00022-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00004-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00010-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00014-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00008-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00020-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00025-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00012-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00017-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00001-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v16-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v16/default/model-00021-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-77693-v16-uploader completed after 174.63s with status: succeeded
Stopping job with name chaiml-kimid-v8b-kimid-77693-v16-uploader
Pipeline stage VLLMUploader completed in 175.04s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.62s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8b-kimid-77693-v16
Waiting for inference service chaiml-kimid-v8b-kimid-77693-v16 to be ready
2026-04-03T07:11:07.792331+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v16
2026-04-03T07:12:07.889200+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v16
2026-04-03T07:13:08.002181+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v16
2026-04-03T07:14:08.144487+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v16
Inference service chaiml-kimid-v8b-kimid-77693-v16 ready after 220.55982613563538s
Pipeline stage VLLMDeployer completed in 221.08s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.454092502593994s
Received healthy response to inference request in 3.5464017391204834s
Received healthy response to inference request in 1.4989285469055176s
Received healthy response to inference request in 1.7966811656951904s
Received healthy response to inference request in 1.508376121520996s
Received healthy response to inference request in 1.4371588230133057s
Received healthy response to inference request in 1.4607446193695068s
Received healthy response to inference request in 1.442613124847412s
Received healthy response to inference request in 1.4978187084197998s
Received healthy response to inference request in 1.5210306644439697s
2026-04-03T07:15:08.239786+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v16
Received healthy response to inference request in 1.4635820388793945s
Received healthy response to inference request in 1.4653503894805908s
Received healthy response to inference request in 1.6874775886535645s
Received healthy response to inference request in 1.5176217555999756s
Received healthy response to inference request in 1.4939830303192139s
Received healthy response to inference request in 1.5967211723327637s
Received healthy response to inference request in 1.7700653076171875s
Received healthy response to inference request in 1.8035554885864258s
Received healthy response to inference request in 1.7933344841003418s
Received healthy response to inference request in 1.5374622344970703s
Received healthy response to inference request in 1.4573369026184082s
Received healthy response to inference request in 1.6204171180725098s
Received healthy response to inference request in 1.6539545059204102s
Received healthy response to inference request in 1.573394775390625s
Received healthy response to inference request in 1.5583837032318115s
Received healthy response to inference request in 1.4637224674224854s
Received healthy response to inference request in 1.5229156017303467s
Received healthy response to inference request in 1.8498892784118652s
Received healthy response to inference request in 1.5764310359954834s
Received healthy response to inference request in 1.5715835094451904s
30 requests
0 failed requests
5th percentile: 1.4492388248443604
10th percentile: 1.460403847694397
20th percentile: 1.4650248050689698
30th percentile: 1.4985955953598022
40th percentile: 1.519667100906372
50th percentile: 1.547922968864441
60th percentile: 1.5746092796325684
70th percentile: 1.6304783344268798
80th percentile: 1.7747191429138185
90th percentile: 1.8081888675689697
95th percentile: 2.7322010517120314
99th percentile: 3.5196320605278015
mean time: 1.7047009468078613
Pipeline stage StressChecker completed in 54.24s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.51s
Shutdown handler de-registered
chaiml-kimid-v8b-kimid_77693_v16 status is now deployed due to DeploymentManager action
chaiml-kimid-v8b-kimid_77693_v16 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v8b-kimid_77693_v16 status is now torndown due to DeploymentManager action