Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-89024-v1-uploader
Waiting for job on chaiml-grpo-glm50-combi-89024-v1-uploader to finish
2026-04-01T03:57:13.258442+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
chaiml-grpo-glm50-combi-89024-v1-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-89024-v1-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-89024-v1-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step300-int4-fix4-intel...
2026-04-01T03:58:13.351193+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T03:59:13.557495+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
chaiml-grpo-glm50-combi-89024-v1-uploader: Downloaded in 161.447s
2026-04-01T04:00:13.699189+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:01:13.809438+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:02:13.925354+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:03:14.279316+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:04:14.379309+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
chaiml-grpo-glm50-combi-89024-v1-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step300-int4-fix4-intel in 462.604s
chaiml-grpo-glm50-combi-89024-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-89024-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-89024-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-89024-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-89024-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-89024-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-89024-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-89024-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-89024-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-89024-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-89024-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-89024-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-89024-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-89024-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-89024-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-89024-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-89024-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-89024-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-89024-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/config.json
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/.gitattributes
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/chat_template.jinja
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/generation_config.json
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/quantization_config.json
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/tokenizer_config.json
2026-04-01T04:05:14.488587+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00081-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00081-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00080-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00080-of-00081.safetensors
2026-04-01T04:06:14.576784+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00011-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00011-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00013-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00013-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00018-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00018-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00044-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00044-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00002-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00002-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00006-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00006-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00005-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00005-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00048-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00048-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00026-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00026-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00053-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00053-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00037-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00037-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00030-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00030-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00027-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00027-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00036-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00036-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00028-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00028-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00012-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00012-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00052-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00052-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00014-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00014-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00020-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00020-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00024-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00024-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00045-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00045-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00016-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00016-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00065-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00065-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00009-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00009-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00054-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00054-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00060-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00060-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00010-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00010-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00022-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00022-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00073-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00073-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00066-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00066-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00023-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00023-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00015-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00015-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00042-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00042-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00008-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00008-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00074-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00074-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00025-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00025-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00061-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00061-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00004-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00004-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00034-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00034-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00067-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00067-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00040-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00040-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00035-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00035-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00007-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00007-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00062-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00062-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00068-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00068-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00017-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00017-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00043-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00043-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00047-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00047-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00077-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00077-of-00081.safetensors
chaiml-grpo-glm50-combi-89024-v1-uploader: cp /dev/shm/model_output/model-00003-of-00081.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-89024-v1/default/model-00003-of-00081.safetensors
Job chaiml-grpo-glm50-combi-89024-v1-uploader completed after 650.24s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-89024-v1-uploader
Pipeline stage VLLMUploader completed in 650.76s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.24s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-89024-v1
Waiting for inference service chaiml-grpo-glm50-combi-89024-v1 to be ready
2026-04-01T04:07:14.663604+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:08:14.771857+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:09:14.875830+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:10:14.998646+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:11:15.366084+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:12:15.474833+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:13:15.579043+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:14:15.706835+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
2026-04-01T04:15:15.811042+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
Inference service chaiml-grpo-glm50-combi-89024-v1 ready after 491.37382102012634s
Pipeline stage VLLMDeployer completed in 491.85s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 10.694380760192871s
Received healthy response to inference request in 2.591085433959961s
Received healthy response to inference request in 2.5805225372314453s
Received healthy response to inference request in 2.5938901901245117s
Received healthy response to inference request in 2.5431723594665527s
Received healthy response to inference request in 2.677431106567383s
Received healthy response to inference request in 2.6091995239257812s
Received healthy response to inference request in 2.6462745666503906s
Received healthy response to inference request in 2.655341625213623s
Received healthy response to inference request in 2.6737747192382812s
Received healthy response to inference request in 2.6245245933532715s
Received healthy response to inference request in 2.6901891231536865s
Received healthy response to inference request in 2.6389384269714355s
Received healthy response to inference request in 2.924041271209717s
Received healthy response to inference request in 2.6463422775268555s
Received healthy response to inference request in 2.507507085800171s
Received healthy response to inference request in 2.647081136703491s
Received healthy response to inference request in 2.5540719032287598s
2026-04-01T04:16:15.952425+00:00 monitor updated for chaiml-grpo-glm50-combi_89024_v1
Received healthy response to inference request in 2.538581371307373s
Received healthy response to inference request in 2.4618799686431885s
Received healthy response to inference request in 2.839026689529419s
Received healthy response to inference request in 2.450446128845215s
Received healthy response to inference request in 2.510197639465332s
Received healthy response to inference request in 2.5133657455444336s
Received healthy response to inference request in 2.495932102203369s
Received healthy response to inference request in 2.4778826236724854s
Received healthy response to inference request in 2.685626268386841s
Received healthy response to inference request in 2.727578639984131s
Received healthy response to inference request in 2.969228506088257s
Received healthy response to inference request in 2.8104617595672607s
30 requests
0 failed requests
5th percentile: 2.469081163406372
10th percentile: 2.4941271543502808
20th percentile: 2.5127321243286134
30th percentile: 2.5508020401000975
40th percentile: 2.5927682876586915
50th percentile: 2.6317315101623535
60th percentile: 2.6466378211975097
70th percentile: 2.674871635437012
80th percentile: 2.6976670265197753
90th percentile: 2.847528147697449
95th percentile: 2.9488942503929136
99th percentile: 8.454086606502539
mean time: 2.8992658694585165
Pipeline stage StressChecker completed in 90.15s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.71s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_89024_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_89024_v1 status is now inactive due to auto deactivation removed underperforming models