Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-23481-v2-uploader
Waiting for job on chaiml-grpo-glm50-combi-23481-v2-uploader to finish
chaiml-grpo-glm50-combi-23481-v2-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-23481-v2-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-23481-v2-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm3331-mega-kimi-mix8020-step200-int4...
2026-04-03T17:02:31.759333+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:03:31.871437+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:04:31.988383+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
chaiml-grpo-glm50-combi-23481-v2-uploader: Downloaded in 167.312s
2026-04-03T17:05:32.090173+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:06:32.208357+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
chaiml-grpo-glm50-combi-23481-v2-uploader: Processed model ChaiML/grpo-glm50-combine-rm3331-mega-kimi-mix8020-step200-int4 in 308.277s
chaiml-grpo-glm50-combi-23481-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-23481-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-23481-v2-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-23481-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-23481-v2-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-23481-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-23481-v2-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-23481-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-23481-v2-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-23481-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-23481-v2-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-23481-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-23481-v2-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-23481-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-23481-v2-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-23481-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-23481-v2-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-23481-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-23481-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/generation_config.json
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/chat_template.jinja
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/.gitattributes
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/config.json
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/tokenizer_config.json
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/quantization_config.json
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model.safetensors.index.json
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/tokenizer.json
2026-04-03T17:07:32.316673+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00075-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00075-of-00075.safetensors
2026-04-03T17:08:32.400255+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00027-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00027-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00007-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00007-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00057-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00057-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00015-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00015-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00009-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00009-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00048-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00048-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00035-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00035-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00003-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00003-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00040-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00040-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00033-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00033-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00008-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00008-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00038-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00038-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00023-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00023-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00042-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00042-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00005-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00005-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00002-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00002-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00053-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00053-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00018-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00018-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00064-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00064-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00013-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00013-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00056-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00056-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00052-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00052-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00016-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00016-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00047-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00047-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00060-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00060-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00026-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00026-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00062-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00062-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00029-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00029-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00030-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00030-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00061-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00061-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00006-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00006-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00065-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00065-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00021-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00021-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00055-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00055-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00020-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00020-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00054-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00054-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00024-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00024-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00046-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00046-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00025-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00025-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00049-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00049-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00063-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00063-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00041-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00041-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00050-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00050-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00043-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00043-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00001-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00001-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00069-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00069-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00045-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00045-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00036-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00036-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00004-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00004-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00070-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00070-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00017-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00017-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00066-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00066-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00037-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00037-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00051-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00051-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00014-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00014-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00019-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00019-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00032-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00032-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00031-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00031-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00039-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00039-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00022-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00022-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00011-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00011-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00028-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00028-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00010-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00010-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00059-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00059-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00068-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00068-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00074-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00074-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00058-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00058-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00044-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00044-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00072-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00072-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00071-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00071-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00012-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00012-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00034-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00034-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00067-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00067-of-00075.safetensors
chaiml-grpo-glm50-combi-23481-v2-uploader: cp /dev/shm/model_output/model-00073-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-23481-v2/default/model-00073-of-00075.safetensors
Job chaiml-grpo-glm50-combi-23481-v2-uploader completed after 450.85s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-23481-v2-uploader
Pipeline stage VLLMUploader completed in 451.37s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.12s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.08s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-23481-v2
Waiting for inference service chaiml-grpo-glm50-combi-23481-v2 to be ready
2026-04-03T17:09:32.516060+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:10:32.664779+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:11:32.783597+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:12:32.905867+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:13:33.040229+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:14:33.179602+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:15:33.307166+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:16:33.423938+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
2026-04-03T17:17:33.538896+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
Inference service chaiml-grpo-glm50-combi-23481-v2 ready after 561.2501873970032s
Pipeline stage VLLMDeployer completed in 561.84s
run pipeline stage %s
Running pipeline stage StressChecker
2026-04-03T17:18:33.667106+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
Received healthy response to inference request in 10.943476438522339s
Received healthy response to inference request in 2.6389715671539307s
Received healthy response to inference request in 11.294245719909668s
Received healthy response to inference request in 10.887970924377441s
Received healthy response to inference request in 2.499818801879883s
Received healthy response to inference request in 2.7199409008026123s
Received healthy response to inference request in 2.7067837715148926s
Received healthy response to inference request in 3.2288126945495605s
Received healthy response to inference request in 2.9028332233428955s
Received healthy response to inference request in 11.213783502578735s
Received healthy response to inference request in 2.6359784603118896s
2026-04-03T17:19:33.765641+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
Received healthy response to inference request in 2.9677538871765137s
Received healthy response to inference request in 2.6628050804138184s
Received healthy response to inference request in 2.7092652320861816s
Received healthy response to inference request in 2.615584373474121s
Received healthy response to inference request in 11.091285467147827s
Received healthy response to inference request in 2.569570302963257s
Received healthy response to inference request in 2.8076696395874023s
Received healthy response to inference request in 2.8673014640808105s
Received healthy response to inference request in 2.5922727584838867s
Received healthy response to inference request in 2.5723392963409424s
Received healthy response to inference request in 2.5073704719543457s
Received healthy response to inference request in 2.705127477645874s
Received healthy response to inference request in 2.4798381328582764s
Received healthy response to inference request in 2.563971757888794s
Received healthy response to inference request in 2.6895558834075928s
2026-04-03T17:20:33.877105+00:00 monitor updated for chaiml-grpo-glm50-combi_23481_v2
Received healthy response to inference request in 2.79504132270813s
Received healthy response to inference request in 2.71722674369812s
Received healthy response to inference request in 2.9412622451782227s
Received healthy response to inference request in 2.717893600463867s
30 requests
0 failed requests
5th percentile: 2.5032170534133913
10th percentile: 2.558311629295349
20th percentile: 2.5882860660552978
30th percentile: 2.6380736351013185
40th percentile: 2.6988988399505613
50th percentile: 2.713245987892151
60th percentile: 2.7499810695648192
70th percentile: 2.8779609918594358
80th percentile: 3.019965648651124
90th percentile: 10.958257341384888
95th percentile: 11.158659386634826
99th percentile: 11.270911676883697
mean time: 4.108191704750061
Pipeline stage StressChecker completed in 136.21s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.66s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_23481_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_23481_v2 status is now inactive due to system request