Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-44700-v3-uploader
Waiting for job on chaiml-grpo-glm50-combi-44700-v3-uploader to finish
chaiml-grpo-glm50-combi-44700-v3-uploader: ChaiML/grpo-glm50-combine-rm3331-mega-kimi-mix8020-step300-int4 is already quantized
chaiml-grpo-glm50-combi-44700-v3-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-44700-v3-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm3331-mega-kimi-mix8020-step300-int4...
2026-04-04T17:59:56.321595+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:00:56.405711+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:01:56.487586+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
chaiml-grpo-glm50-combi-44700-v3-uploader: Downloaded in 161.447s
chaiml-grpo-glm50-combi-44700-v3-uploader: Processed model ChaiML/grpo-glm50-combine-rm3331-mega-kimi-mix8020-step300-int4 in 161.697s
chaiml-grpo-glm50-combi-44700-v3-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-44700-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-44700-v3-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-44700-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-44700-v3-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-44700-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-44700-v3-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-44700-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-44700-v3-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-44700-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-44700-v3-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-44700-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-44700-v3-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-44700-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-44700-v3-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-44700-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-44700-v3-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-44700-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-44700-v3-uploader: uploading /tmp/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/.gitattributes
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/generation_config.json
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/chat_template.jinja
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/config.json
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/quantization_config.json
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/tokenizer_config.json
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model.safetensors.index.json
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/tokenizer.json
2026-04-04T18:02:56.574743+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00075-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00075-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00029-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00029-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00042-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00042-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00030-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00030-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00008-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00008-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00065-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00065-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00074-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00074-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00057-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00057-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00071-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00071-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00031-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00031-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00066-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00066-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00003-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00003-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00048-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00048-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00058-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00058-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00005-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00005-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00025-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00025-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00013-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00013-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00051-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00051-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00022-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00022-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00012-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00012-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00046-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00046-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00041-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00041-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00039-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00039-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00063-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00063-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00067-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00067-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00054-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00054-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00021-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00021-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00002-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00002-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00049-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00049-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00016-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00016-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00070-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00070-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00050-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00050-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00019-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00019-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00069-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00069-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00035-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00035-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00060-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00060-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00045-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00045-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00006-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00006-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00068-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00068-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00055-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00055-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00010-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00010-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00047-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00047-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00007-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00007-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00026-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00026-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00056-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00056-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00072-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00072-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00073-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00073-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00062-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00062-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00027-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00027-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00037-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00037-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00023-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00023-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00064-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00064-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00028-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00028-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00014-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00014-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00032-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00032-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00059-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00059-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00015-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00015-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00038-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00038-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00034-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00034-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00020-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00020-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00036-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00036-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00001-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00001-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00017-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00017-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00053-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00053-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00052-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00052-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00011-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00011-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00009-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00009-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00044-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00044-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00004-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00004-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00040-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00040-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00033-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00033-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00018-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00018-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00024-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00024-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00061-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00061-of-00075.safetensors
chaiml-grpo-glm50-combi-44700-v3-uploader: cp /tmp/model_output/model-00043-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-44700-v3/default/model-00043-of-00075.safetensors
2026-04-04T18:03:56.662524+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
Job chaiml-grpo-glm50-combi-44700-v3-uploader completed after 308.38s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-44700-v3-uploader
Pipeline stage VLLMUploader completed in 309.47s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.09s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-44700-v3
Waiting for inference service chaiml-grpo-glm50-combi-44700-v3 to be ready
2026-04-04T18:04:56.762893+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:05:57.374778+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:06:57.492546+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:07:57.638360+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:08:57.735924+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:09:57.866024+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:10:58.061364+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:11:58.172878+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:12:58.280067+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:13:58.388521+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:14:58.545542+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:15:58.645071+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:16:58.738939+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:17:58.828967+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:18:58.931389+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:19:59.027991+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
2026-04-04T18:20:59.129395+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
Inference service chaiml-grpo-glm50-combi-44700-v3 ready after 1033.8820288181305s
Pipeline stage VLLMDeployer completed in 1034.42s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 10.85713505744934s
Received healthy response to inference request in 2.68005633354187s
Received healthy response to inference request in 10.906170129776001s
Received healthy response to inference request in 2.57463002204895s
Received healthy response to inference request in 2.602301597595215s
Received healthy response to inference request in 2.6822404861450195s
Received healthy response to inference request in 2.5866141319274902s
2026-04-04T18:21:59.223340+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
Received healthy response to inference request in 2.6162142753601074s
Received healthy response to inference request in 2.761651039123535s
Received healthy response to inference request in 2.67862868309021s
Received healthy response to inference request in 2.80151104927063s
Received healthy response to inference request in 2.976776599884033s
Received healthy response to inference request in 3.0113983154296875s
Received healthy response to inference request in 2.6192703247070312s
Received healthy response to inference request in 2.8047032356262207s
Received healthy response to inference request in 2.722879648208618s
Received healthy response to inference request in 2.872833728790283s
Received healthy response to inference request in 2.8819010257720947s
Received healthy response to inference request in 2.566070079803467s
Received healthy response to inference request in 2.801922082901001s
Received healthy response to inference request in 2.942548990249634s
Received healthy response to inference request in 2.687242031097412s
Received healthy response to inference request in 2.5661914348602295s
Received healthy response to inference request in 2.5857722759246826s
Received healthy response to inference request in 2.71679425239563s
Received healthy response to inference request in 2.635993719100952s
Received healthy response to inference request in 2.6225736141204834s
Received healthy response to inference request in 2.8549282550811768s
2026-04-04T18:22:59.328334+00:00 monitor updated for chaiml-grpo-glm50-combi_44700_v3
Received healthy response to inference request in 2.672795057296753s
Received healthy response to inference request in 3.2662267684936523s
30 requests
0 failed requests
5th percentile: 2.569988799095154
10th percentile: 2.5846580505371093
20th percentile: 2.6134317398071287
30th percentile: 2.6319676876068114
40th percentile: 2.6794852733612062
50th percentile: 2.702018141746521
60th percentile: 2.777595043182373
70th percentile: 2.8197707414627073
80th percentile: 2.894030618667603
90th percentile: 3.036881160736084
95th percentile: 7.44122632741926
99th percentile: 10.89194995880127
mean time: 3.2851991415023805
Pipeline stage StressChecker completed in 102.00s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_44700_v3 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_44700_v3 status is now inactive due to auto deactivation removed underperforming models