Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm46-combi-19477-v2-uploader
Waiting for job on chaiml-grpo-glm46-combi-19477-v2-uploader to finish
chaiml-grpo-glm46-combi-19477-v2-uploader: Using quantization_mode: fp8
chaiml-grpo-glm46-combi-19477-v2-uploader: Checking if ChaiML/grpo-glm46-combine-rm82-lr5e6-step500-FP8 already exists in ChaiML
chaiml-grpo-glm46-combi-19477-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-glm46-combi-19477-v2-uploader: Downloading snapshot of ChaiML/grpo-glm46-combine-rm82-lr5e6-step500-FP8...
2026-03-12T16:21:02.325092+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm46-combin-2308-v2-uploader
Waiting for job on chaiml-gspo-glm46-combin-2308-v2-uploader to finish
2026-03-12T16:22:02.447956+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
chaiml-gspo-glm46-combin-2308-v2-uploader: Checking if ChaiML/gspo-glm46-combine-rm55-step100-FP8 already exists in ChaiML
chaiml-gspo-glm46-combin-2308-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-gspo-glm46-combin-2308-v2-uploader: Downloading snapshot of ChaiML/gspo-glm46-combine-rm55-step100-FP8...
2026-03-12T16:22:37.647262+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
2026-03-12T16:23:02.589516+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
chaiml-grpo-glm46-combi-19477-v2-uploader: Downloaded in 160.474s
chaiml-grpo-glm46-combi-19477-v2-uploader: Processed model ChaiML/grpo-glm46-combine-rm82-lr5e6-step500 in 163.958s
chaiml-grpo-glm46-combi-19477-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm46-combi-19477-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-19477-v2-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm46-combi-19477-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm46-combi-19477-v2-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm46-combi-19477-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-19477-v2-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-19477-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-19477-v2-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm46-combi-19477-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
2026-03-12T16:23:37.788745+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
chaiml-grpo-glm46-combi-19477-v2-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-19477-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm46-combi-19477-v2-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm46-combi-19477-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm46-combi-19477-v2-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm46-combi-19477-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm46-combi-19477-v2-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm46-combi-19477-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm46-combi-19477-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/.gitattributes
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/generation_config.json
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/chat_template.jinja
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/config.json
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/special_tokens_map.json
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/recipe.yaml
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/tokenizer_config.json
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model.safetensors.index.json
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/tokenizer.json
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00072-of-00072.safetensors
2026-03-12T16:24:02.739295+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00071-of-00072.safetensors
2026-03-12T16:24:38.012464+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00033-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00027-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00010-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00048-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00026-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00020-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00068-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00046-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00028-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00008-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00007-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00034-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00024-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00029-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00013-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00011-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00037-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00031-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00012-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00058-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00002-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00045-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00065-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00044-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00001-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00035-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00043-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00069-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00032-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00055-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00005-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00030-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00059-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00019-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00023-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00004-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00040-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00063-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00016-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00053-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00025-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00049-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00006-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00054-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00038-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00060-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00018-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00064-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00047-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00051-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00057-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00021-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00003-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00009-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00070-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00067-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00061-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00015-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00041-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00036-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00022-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00066-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00050-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00039-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00042-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00056-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00017-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00052-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00014-of-00072.safetensors
chaiml-grpo-glm46-combi-19477-v2-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm46-combi-19477-v2/default/model-00062-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: Downloaded in 158.020s
chaiml-gspo-glm46-combin-2308-v2-uploader: Processed model ChaiML/gspo-glm46-combine-rm55-step100 in 161.607s
2026-03-12T16:25:02.991926+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
Job chaiml-grpo-glm46-combi-19477-v2-uploader completed after 304.77s with status: succeeded
Stopping job with name chaiml-grpo-glm46-combi-19477-v2-uploader
Pipeline stage VLLMUploader completed in 306.07s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.18s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm46-combi-19477-v2
chaiml-gspo-glm46-combin-2308-v2-uploader: creating bucket guanaco-vllm-models
Waiting for inference service chaiml-grpo-glm46-combi-19477-v2 to be ready
chaiml-gspo-glm46-combin-2308-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combin-2308-v2-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm46-combin-2308-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm46-combin-2308-v2-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm46-combin-2308-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combin-2308-v2-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combin-2308-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combin-2308-v2-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm46-combin-2308-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combin-2308-v2-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combin-2308-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm46-combin-2308-v2-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm46-combin-2308-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm46-combin-2308-v2-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm46-combin-2308-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm46-combin-2308-v2-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm46-combin-2308-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm46-combin-2308-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/.gitattributes
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/generation_config.json
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/tokenizer_config.json
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model.safetensors.index.json
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/chat_template.jinja
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/config.json
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/tokenizer.json
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/recipe.yaml
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/special_tokens_map.json
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00072-of-00072.safetensors
2026-03-12T16:25:38.504782+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00071-of-00072.safetensors
2026-03-12T16:26:03.466708+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00002-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00070-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00056-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00058-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00005-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00064-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00047-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00020-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00029-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00026-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00052-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00001-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00006-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00025-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00031-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00028-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00019-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00055-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00016-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00063-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00037-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00046-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00039-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00012-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00034-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00060-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00035-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00040-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00041-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00048-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00066-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00067-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00011-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00013-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00008-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00018-of-00072.safetensors
chaiml-gspo-glm46-combin-2308-v2-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm46-combin-2308-v2/default/model-00003-of-00072.safetensors
Job chaiml-gspo-glm46-combin-2308-v2-uploader completed after 298.13s with status: succeeded
Stopping job with name chaiml-gspo-glm46-combin-2308-v2-uploader
Pipeline stage VLLMUploader completed in 298.93s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.09s
run pipeline stage %s
Running pipeline stage VLLMDeployer
2026-03-12T16:26:38.666250+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
Creating inference service chaiml-gspo-glm46-combin-2308-v2
Waiting for inference service chaiml-gspo-glm46-combin-2308-v2 to be ready
2026-03-12T16:27:03.628702+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
2026-03-12T16:27:38.855540+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
2026-03-12T16:28:03.819418+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
2026-03-12T16:28:39.027165+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
2026-03-12T16:29:03.988096+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
2026-03-12T16:29:39.236231+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
2026-03-12T16:30:04.180894+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
2026-03-12T16:30:39.438864+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
Inference service chaiml-grpo-glm46-combi-19477-v2 ready after 351.84844040870667s
Pipeline stage VLLMDeployer completed in 352.84s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-12T16:31:04.371652+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
Received healthy response to inference request in 9.258665084838867s
Received healthy response to inference request in 9.332569599151611s
Received healthy response to inference request in 1.9796473979949951s
Received healthy response to inference request in 8.913277626037598s
Received healthy response to inference request in 2.1534109115600586s
Received healthy response to inference request in 1.9950776100158691s
2026-03-12T16:31:39.607609+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
Received healthy response to inference request in 2.1589479446411133s
Received healthy response to inference request in 8.949825286865234s
Received healthy response to inference request in 9.175184488296509s
Received healthy response to inference request in 1.9742887020111084s
2026-03-12T16:32:04.554184+00:00 monitor updated for chaiml-grpo-glm46-combi_19477_v2
Received healthy response to inference request in 2.127040147781372s
Received healthy response to inference request in 2.013502836227417s
Received healthy response to inference request in 2.090557813644409s
Received healthy response to inference request in 2.1972997188568115s
Received healthy response to inference request in 2.190857172012329s
Received healthy response to inference request in 2.0132908821105957s
Received healthy response to inference request in 2.1784110069274902s
Received healthy response to inference request in 2.246751308441162s
Received healthy response to inference request in 2.1196861267089844s
Received healthy response to inference request in 2.004427909851074s
Received healthy response to inference request in 2.028691530227661s
Received healthy response to inference request in 2.1131627559661865s
Received healthy response to inference request in 2.029627799987793s
Received healthy response to inference request in 2.282078266143799s
2026-03-12T16:32:39.919376+00:00 monitor updated for chaiml-gspo-glm46-combin_2308_v2
Inference service chaiml-gspo-glm46-combin-2308-v2 ready after 361.4945635795593s
Pipeline stage VLLMDeployer completed in 362.42s
Received healthy response to inference request in 2.2171075344085693s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3072662353515625s
Received healthy response to inference request in 2.0240046977996826s
Received healthy response to inference request in 2.0804340839385986s
Received healthy response to inference request in 2.0486464500427246s
Received healthy response to inference request in 9.054048299789429s
Received healthy response to inference request in 2.2596614360809326s
30 requests
0 failed requests
5th percentile: 1.9865909934043884
10th percentile: 2.0034928798675535
20th percentile: 2.0219043254852296
30th percentile: 2.042940855026245
40th percentile: 2.1041207790374754
50th percentile: 2.1402255296707153
60th percentile: 2.1833894729614256
70th percentile: 2.226000666618347
80th percentile: 2.287115859985352
90th percentile: 8.972361207008362
95th percentile: 9.221098816394806
99th percentile: 9.311137290000916
mean time: 3.2821133454640705
Pipeline stage StressChecker completed in 113.05s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.10s
Shutdown handler de-registered
chaiml-grpo-glm46-combi_19477_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-glm46-combi_19477_v2 status is now inactive due to auto deactivation removed underperforming models