chaiml-grpo-glm50-combi_78846

developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-combi_78846_v2
model_name: chaiml-grpo-glm50-combi_78846_v2
model_group: ChaiML/grpo-glm50-combin
status: torndown
timestamp: 2026-04-01T01:21:56+00:00
num_battles: 10806
num_wins: 6451
celo_rating: 1364.05
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_78846_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast
model_size: 31B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5969831575050898
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '####', '<|user|>', '<|im_end|>', '<|assistant|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-78846-v2-uploader
Waiting for job on chaiml-grpo-glm50-combi-78846-v2-uploader to finish
chaiml-grpo-glm50-combi-78846-v2-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-78846-v2-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-78846-v2-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast...
2026-03-28T22:20:21.283481+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:21:21.374062+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
chaiml-grpo-glm50-combi-78846-v2-uploader: Downloaded in 123.379s
2026-03-28T22:22:21.462852+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:23:21.598370+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:24:21.694002+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:25:21.829024+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
chaiml-grpo-glm50-combi-78846-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-78846-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-78846-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-78846-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-78846-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-78846-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-78846-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-78846-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-78846-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-78846-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-78846-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-78846-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-78846-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-78846-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/.gitattributes
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/chat_template.jinja
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/generation_config.json
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/tokenizer_config.json
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/quantization_config.json
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/config.json
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model.safetensors.index.json
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/tokenizer.json
2026-03-28T22:26:27.433580+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00077-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00077-of-00078.safetensors
Failed to get response for submission chaiml-pony-d3b-mv1-wi_84391_v10: ('http://chaiml-pony-d3b-mv1-wi-84391-v10-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'request timeout')
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00078-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00078-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00051-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00051-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00032-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00032-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00022-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00022-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00030-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00030-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00001-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00001-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00066-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00066-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00053-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00053-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00038-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00038-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00031-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00031-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00064-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00064-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00012-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00012-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00017-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00017-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00033-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00033-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00003-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00003-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00058-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00058-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00054-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00054-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00065-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00065-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00069-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00069-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00013-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00013-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00039-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00039-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00052-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00052-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00010-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00010-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00040-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00040-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00046-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00046-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00014-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00014-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00050-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00050-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00027-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00027-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00006-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00006-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00067-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00067-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00024-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00024-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00070-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00070-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00073-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00073-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00062-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00062-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00076-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00076-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00060-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00060-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00036-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00036-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00075-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00075-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00043-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00043-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00028-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00028-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00019-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00019-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00009-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00009-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00023-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00023-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00008-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00008-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00068-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00068-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00056-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00056-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00020-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00020-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00037-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00037-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00071-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00071-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00049-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00049-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00005-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00005-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00025-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00025-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00026-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00026-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00057-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00057-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00018-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00018-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00044-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00044-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00048-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00048-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00011-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00011-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00055-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00055-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00063-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00063-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00004-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00004-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00002-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00002-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00034-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00034-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00059-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00059-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00061-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00061-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00015-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00015-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00029-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00029-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00041-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00041-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00074-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00074-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00021-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00021-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00007-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00007-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00047-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00047-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00072-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00072-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00045-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00045-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00042-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00042-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v2-uploader: cp /dev/shm/model_output/model-00035-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v2/default/model-00035-of-00078.safetensors
2026-03-28T22:27:27.523699+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
Job chaiml-grpo-glm50-combi-78846-v2-uploader completed after 489.61s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-78846-v2-uploader
Pipeline stage VLLMUploader completed in 490.09s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.44s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-78846-v2
Waiting for inference service chaiml-grpo-glm50-combi-78846-v2 to be ready
2026-03-28T22:28:27.629450+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:29:27.726747+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:30:27.822976+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:31:27.927521+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:32:28.031955+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:33:28.128583+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:34:28.222373+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
2026-03-28T22:35:28.321293+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
Inference service chaiml-grpo-glm50-combi-78846-v2 ready after 480.9073326587677s
Pipeline stage VLLMDeployer completed in 481.40s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 10.886971950531006s
Received healthy response to inference request in 2.544987201690674s
Received healthy response to inference request in 2.5912508964538574s
Received healthy response to inference request in 10.790597200393677s
Received healthy response to inference request in 2.421353816986084s
Received healthy response to inference request in 2.5131213665008545s
Received healthy response to inference request in 3.1197917461395264s
Received healthy response to inference request in 2.589801549911499s
Received healthy response to inference request in 2.8150508403778076s
Received healthy response to inference request in 2.8504998683929443s
Received healthy response to inference request in 2.457240104675293s
Received healthy response to inference request in 2.770552158355713s
2026-03-28T22:36:28.414748+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v2
Received healthy response to inference request in 2.5214362144470215s
Received healthy response to inference request in 2.6067874431610107s
Received healthy response to inference request in 2.5595173835754395s
Received healthy response to inference request in 2.474691152572632s
Received healthy response to inference request in 2.735185384750366s
Received healthy response to inference request in 2.5081846714019775s
Received healthy response to inference request in 2.4902782440185547s
Received healthy response to inference request in 2.535688638687134s
Received healthy response to inference request in 2.5328691005706787s
Received healthy response to inference request in 2.816282033920288s
Received healthy response to inference request in 2.848076343536377s
Received healthy response to inference request in 2.4590182304382324s
Received healthy response to inference request in 2.7327373027801514s
Received healthy response to inference request in 2.4417152404785156s
Received healthy response to inference request in 2.586949110031128s
Received healthy response to inference request in 2.5307798385620117s
Received healthy response to inference request in 2.6283299922943115s
Received healthy response to inference request in 2.808493137359619s
30 requests
0 failed requests
5th percentile: 2.4487014293670653
10th percentile: 2.4588404178619383
20th percentile: 2.504603385925293
30th percentile: 2.5279767513275146
40th percentile: 2.5412677764892577
50th percentile: 2.5883753299713135
60th percentile: 2.615404462814331
70th percentile: 2.74579541683197
80th percentile: 2.815297079086304
90th percentile: 2.877429056167603
95th percentile: 7.338734745979288
99th percentile: 10.85902327299118
mean time: 3.172274605433146
Pipeline stage StressChecker completed in 99.10s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.29s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_78846_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_78846_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm50-combi_78846_v2 status is now torndown due to DeploymentManager action