Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm4-rm-ck600-ava-v2-uploader
Waiting for job on chaiml-glm4-rm-ck600-ava-v2-uploader to finish
chaiml-glm4-rm-ck600-ava-v2-uploader: Using quantization_mode: none
chaiml-glm4-rm-ck600-ava-v2-uploader: Downloading snapshot of ChaiML/GLM4-RM-CK600-AVA...
2026-04-14T01:07:29.721835+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
chaiml-glm4-rm-ck600-ava-v2-uploader: Downloaded in 52.422s
chaiml-glm4-rm-ck600-ava-v2-uploader: Processed model ChaiML/GLM4-RM-CK600-AVA in 52.564s
chaiml-glm4-rm-ck600-ava-v2-uploader: creating bucket guanaco-vllm-models
chaiml-glm4-rm-ck600-ava-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm4-rm-ck600-ava-v2-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm4-rm-ck600-ava-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm4-rm-ck600-ava-v2-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm4-rm-ck600-ava-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm4-rm-ck600-ava-v2-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm4-rm-ck600-ava-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm4-rm-ck600-ava-v2-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm4-rm-ck600-ava-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm4-rm-ck600-ava-v2-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm4-rm-ck600-ava-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm4-rm-ck600-ava-v2-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm4-rm-ck600-ava-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm4-rm-ck600-ava-v2-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm4-rm-ck600-ava-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm4-rm-ck600-ava-v2-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm4-rm-ck600-ava-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm4-rm-ck600-ava-v2-uploader: uploading /tmp/model_output to s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/chat_template.jinja
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/.gitattributes
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/model.safetensors.index.json
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/zero_to_fp32.py s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/zero_to_fp32.py
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/latest s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/latest
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/config.json s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/config.json
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/tokenizer_config.json
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/tokenizer.json
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/model-00002-of-00002.safetensors
2026-04-14T01:08:29.811787+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
chaiml-glm4-rm-ck600-ava-v2-uploader: cp /tmp/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-glm4-rm-ck600-ava-v2/default/model-00001-of-00002.safetensors
Job chaiml-glm4-rm-ck600-ava-v2-uploader completed after 164.02s with status: succeeded
Stopping job with name chaiml-glm4-rm-ck600-ava-v2-uploader
Pipeline stage VLLMUploader completed in 164.48s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 6.68s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm4-rm-ck600-ava-v2
Waiting for inference service chaiml-glm4-rm-ck600-ava-v2 to be ready
2026-04-14T01:09:29.901823+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:10:29.988208+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:11:30.076118+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:12:30.172637+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:13:30.283143+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:14:30.465930+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:15:30.559832+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:16:30.660162+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:17:30.866306+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:18:30.957609+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:19:31.056518+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:20:31.146686+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
2026-04-14T01:21:31.238624+00:00 monitor updated for chaiml-glm4-rm-ck600-ava_v2
Inference service chaiml-glm4-rm-ck600-ava-v2 ready after 753.8308379650116s
Pipeline stage VLLMDeployer completed in 754.41s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.319289207458496s
Received healthy response to inference request in 3.269306182861328s
Received healthy response to inference request in 4.03146767616272s
Received healthy response to inference request in 3.819955587387085s
Received healthy response to inference request in 3.0236644744873047s
5 requests
0 failed requests
5th percentile: 3.072792816162109
10th percentile: 3.121921157836914
20th percentile: 3.2201778411865236
30th percentile: 3.3794360637664793
40th percentile: 3.599695825576782
50th percentile: 3.819955587387085
60th percentile: 3.904560422897339
70th percentile: 3.989165258407593
80th percentile: 4.089031982421875
90th percentile: 4.204160594940186
95th percentile: 4.261724901199341
99th percentile: 4.307776346206665
mean time: 3.6927366256713867
Pipeline stage StressChecker completed in 19.89s
Shutdown handler de-registered
chaiml-glm4-rm-ck600-ava_v2 status is now deployed due to DeploymentManager action
chaiml-glm4-rm-ck600-ava_v2 status is now inactive due to auto deactivation removed underperforming models