developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-combi_78846_v4
model_name: chaiml-grpo-glm50-combi_78846_v4
model_group: ChaiML/grpo-glm50-combin
status: deployed
timestamp: 2026-04-01T18:22:28+00:00
num_battles: 9518
num_wins: 5537
celo_rating: 8398.48
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_78846_v4
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast
model_size: 31B
ranking_group: single
us_pacific_date: 2026-04-01
win_ratio: 0.5817398613154023
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|im_end|>', '####', '</s>', '<|user|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-78846-v4-uploader
Waiting for job on chaiml-grpo-glm50-combi-78846-v4-uploader to finish
chaiml-grpo-glm50-combi-78846-v4-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-78846-v4-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-78846-v4-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast...
2026-04-01T17:54:08.281182+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T17:55:08.364351+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T17:56:08.446011+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
chaiml-grpo-glm50-combi-78846-v4-uploader: Downloaded in 182.719s
2026-04-01T17:57:08.565549+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Retrying (%r) after connection broken by '%r': %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-01T17:58:08.664254+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Retrying (%r) after connection broken by '%r': %s
2026-04-01T17:59:08.767670+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
chaiml-grpo-glm50-combi-78846-v4-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast in 322.516s
chaiml-grpo-glm50-combi-78846-v4-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-78846-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-78846-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-78846-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-78846-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-78846-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-78846-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-78846-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-78846-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-78846-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-78846-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-78846-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-78846-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-78846-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/.gitattributes
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/generation_config.json
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/chat_template.jinja
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/config.json
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/tokenizer_config.json
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/quantization_config.json
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model.safetensors.index.json
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/tokenizer.json
2026-04-01T18:00:09.384515+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00078-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00078-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00077-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00077-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00005-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00005-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00071-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00071-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00054-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00054-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00048-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00048-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00067-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00067-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00061-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00061-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00030-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00030-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00025-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00025-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00018-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00018-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00008-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00008-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00059-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00059-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00076-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00076-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00012-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00012-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00042-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00042-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00031-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00031-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00040-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00040-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00020-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00020-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00053-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00053-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00056-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00056-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00032-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00032-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00033-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00033-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00026-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00026-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00050-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00050-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00002-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00002-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00035-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00035-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00049-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00049-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00037-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00037-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00072-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00072-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00062-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00062-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00065-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00065-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00074-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00074-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00009-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00009-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00027-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00027-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00039-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00039-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00047-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00047-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00044-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00044-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00055-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00055-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00022-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00022-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00003-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00003-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00010-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00010-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00046-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00046-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00017-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00017-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00052-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00052-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00070-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00070-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00063-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00063-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00038-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00038-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00006-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00006-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00024-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00024-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00021-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00021-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00007-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00007-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00073-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00073-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00015-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00015-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00019-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00019-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00041-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00041-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00060-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00060-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00036-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00036-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00001-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00001-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00013-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00013-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00011-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00011-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00043-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00043-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00075-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00075-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00023-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00023-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00004-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00004-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00034-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00034-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v4-uploader: cp /dev/shm/model_output/model-00029-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v4/default/model-00029-of-00078.safetensors
2026-04-01T18:01:09.544811+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Job chaiml-grpo-glm50-combi-78846-v4-uploader completed after 482.78s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-78846-v4-uploader
Pipeline stage VLLMUploader completed in 483.34s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.13s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.45s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-78846-v4
Waiting for inference service chaiml-grpo-glm50-combi-78846-v4 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-01T18:02:09.745198+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:03:09.918259+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:04:10.064130+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:05:10.284417+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:06:10.516624+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:07:10.734266+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:08:11.260983+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Retrying (%r) after connection broken by '%r': %s
2026-04-01T18:09:11.473172+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Retrying (%r) after connection broken by '%r': %s
2026-04-01T18:10:11.689535+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:11:16.490048+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:12:16.763230+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-01T18:13:16.893115+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:14:16.997110+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:15:17.122348+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-01T18:16:17.232872+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-01T18:17:17.365627+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:18:17.469401+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
2026-04-01T18:19:17.794911+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-01T18:20:18.019199+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Inference service chaiml-grpo-glm50-combi-78846-v4 ready after 1173.92968416214s
Pipeline stage VLLMDeployer completed in 1174.45s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 11.53567886352539s
Received healthy response to inference request in 2.6783645153045654s
Received healthy response to inference request in 2.605210542678833s
Received healthy response to inference request in 2.503887176513672s
Received healthy response to inference request in 2.566365957260132s
Received healthy response to inference request in 2.76180362701416s
Received healthy response to inference request in 3.3211121559143066s
2026-04-01T18:21:18.171873+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Received healthy response to inference request in 2.9862494468688965s
Received healthy response to inference request in 3.4200592041015625s
Received healthy response to inference request in 2.6093647480010986s
Received healthy response to inference request in 2.5446178913116455s
Received healthy response to inference request in 3.157135486602783s
Received healthy response to inference request in 2.9750120639801025s
Received healthy response to inference request in 2.610837459564209s
Received healthy response to inference request in 2.8421642780303955s
Received healthy response to inference request in 2.6769139766693115s
Received healthy response to inference request in 2.9594995975494385s
Received healthy response to inference request in 3.1154112815856934s
Received healthy response to inference request in 2.882715940475464s
Received healthy response to inference request in 2.7401742935180664s
Received healthy response to inference request in 3.1651477813720703s
Received healthy response to inference request in 2.566291093826294s
Received healthy response to inference request in 2.9725303649902344s
Received healthy response to inference request in 2.5924251079559326s
Received healthy response to inference request in 2.7980055809020996s
Received healthy response to inference request in 3.002373218536377s
Received healthy response to inference request in 2.744919538497925s
2026-04-01T18:22:18.342092+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v4
Received healthy response to inference request in 2.8119590282440186s
Received healthy response to inference request in 2.962252140045166s
Received healthy response to inference request in 2.825730800628662s
30 requests
0 failed requests
5th percentile: 2.5543708324432375
10th percentile: 2.566358470916748
20th percentile: 2.6085339069366453
30th percentile: 2.6779293537139894
40th percentile: 2.755049991607666
50th percentile: 2.8188449144363403
60th percentile: 2.9134294033050536
70th percentile: 2.9732748746871946
80th percentile: 3.0249808311462405
90th percentile: 3.180744218826294
95th percentile: 3.375533032417297
99th percentile: 9.182149162292488
mean time: 3.131140438715617
Pipeline stage StressChecker completed in 99.96s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.91s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_78846_v4 status is now deployed due to DeploymentManager action