chaiml-grpo-glm50-combi_78846

developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-combi_78846_v1
model_name: chaiml-grpo-glm50-combi_78846_v1
model_group: ChaiML/grpo-glm50-combin
status: torndown
timestamp: 2026-03-29T23:41:44+00:00
num_battles: 10639
num_wins: 6293
celo_rating: 1360.19
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_78846_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast
model_size: 31B
ranking_group: single
us_pacific_date: 2026-03-26
win_ratio: 0.5915029608045869
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|user|>', '<|im_end|>', '####', '<|assistant|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-78846-v1-uploader
Waiting for job on chaiml-grpo-glm50-combi-78846-v1-uploader to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-26T21:22:36.517530+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
chaiml-grpo-glm50-combi-78846-v1-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-78846-v1-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast...
Retrying (%r) after connection broken by '%r': %s
2026-03-26T21:23:37.036043+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-26T21:24:37.213302+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
chaiml-grpo-glm50-combi-78846-v1-uploader: Downloaded in 150.075s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-26T21:25:37.306558+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-26T21:26:37.401363+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-26T21:27:37.487228+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
chaiml-grpo-glm50-combi-78846-v1-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step240-int4-fast in 308.977s
chaiml-grpo-glm50-combi-78846-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-78846-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-78846-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-78846-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-78846-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-78846-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-78846-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-78846-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-78846-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-78846-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-78846-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-78846-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-78846-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-78846-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-78846-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/config.json
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/generation_config.json
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/chat_template.jinja
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/.gitattributes
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/tokenizer_config.json
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/quantization_config.json
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/tokenizer.json
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model.safetensors.index.json
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-26T21:28:37.577690+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00077-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00077-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00078-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00078-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00032-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00032-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00002-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00002-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00030-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00030-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00044-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00044-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00007-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00007-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00036-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00036-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00039-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00039-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00027-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00027-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00028-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00028-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00070-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00070-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00047-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00047-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00040-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00040-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00011-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00011-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00001-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00001-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00008-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00008-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00019-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00019-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00014-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00014-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00026-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00026-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00045-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00045-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00022-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00022-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00033-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00033-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00029-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00029-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00005-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00005-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00048-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00048-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00038-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00038-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00024-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00024-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00006-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00006-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00020-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00020-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00073-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00073-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00004-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00004-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00031-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00031-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00021-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00021-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00067-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00067-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00012-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00012-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00051-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00051-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00071-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00071-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00055-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00055-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00068-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00068-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00010-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00010-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00075-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00075-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00034-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00034-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00043-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00043-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00061-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00061-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00013-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00013-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00035-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00035-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00046-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00046-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00009-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00009-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00041-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00041-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00042-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00042-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00066-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00066-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00060-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00060-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00072-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00072-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00069-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00069-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00023-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00023-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00074-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00074-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00025-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00025-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00015-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00015-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00003-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00003-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00058-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00058-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00054-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00054-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00017-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00017-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00059-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00059-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00057-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00057-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00049-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00049-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00018-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00018-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00064-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00064-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00052-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00052-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00037-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00037-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00016-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00016-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00076-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00076-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00065-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00065-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00053-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00053-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00056-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00056-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00063-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00063-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00062-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00062-of-00078.safetensors
chaiml-grpo-glm50-combi-78846-v1-uploader: cp /dev/shm/model_output/model-00050-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-78846-v1/default/model-00050-of-00078.safetensors
Failed to get request counts for guanaco-submitter. Falling back to default
Job chaiml-grpo-glm50-combi-78846-v1-uploader completed after 471.61s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-78846-v1-uploader
Pipeline stage VLLMUploader completed in 472.14s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.73s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-78846-v1
Waiting for inference service chaiml-grpo-glm50-combi-78846-v1 to be ready
2026-03-26T21:29:37.670979+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-26T21:30:37.779361+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
2026-03-26T21:31:41.129572+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
2026-03-26T21:32:41.234275+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
2026-03-26T21:33:41.334446+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
2026-03-26T21:34:41.716439+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-26T21:35:41.897228+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
Inference service chaiml-grpo-glm50-combi-78846-v1 ready after 391.1157183647156s
Pipeline stage VLLMDeployer completed in 391.67s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 10.9864022731781s
Received healthy response to inference request in 2.4077510833740234s
Received healthy response to inference request in 2.5138304233551025s
Received healthy response to inference request in 11.298572301864624s
Received healthy response to inference request in 3.5532827377319336s
2026-03-26T21:36:41.999099+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
Received healthy response to inference request in 10.921454429626465s
Received healthy response to inference request in 2.497006893157959s
Received healthy response to inference request in 11.129652738571167s
Received healthy response to inference request in 2.5170633792877197s
Received healthy response to inference request in 2.499825954437256s
Received healthy response to inference request in 2.8631017208099365s
Received healthy response to inference request in 2.6386427879333496s
Received healthy response to inference request in 2.5806665420532227s
Received healthy response to inference request in 2.9844911098480225s
Received healthy response to inference request in 2.7081172466278076s
Received healthy response to inference request in 2.5730245113372803s
Received healthy response to inference request in 2.5433058738708496s
Received healthy response to inference request in 2.5546669960021973s
Received healthy response to inference request in 2.535588264465332s
Received healthy response to inference request in 2.605499029159546s
Received healthy response to inference request in 2.8905062675476074s
Received healthy response to inference request in 2.4724180698394775s
Received healthy response to inference request in 2.8477981090545654s
Received healthy response to inference request in 2.5219008922576904s
2026-03-26T21:37:42.111767+00:00 monitor updated for chaiml-grpo-glm50-combi_78846_v1
Received healthy response to inference request in 2.6412034034729004s
Received healthy response to inference request in 2.4753973484039307s
Received healthy response to inference request in 2.4977409839630127s
Received healthy response to inference request in 2.5336928367614746s
Received healthy response to inference request in 2.7274210453033447s
Received healthy response to inference request in 2.9320499897003174s
30 requests
0 failed requests
5th percentile: 2.4737587451934813
10th percentile: 2.4948459386825563
20th percentile: 2.511029529571533
30th percentile: 2.5301552534103395
40th percentile: 2.550122547149658
50th percentile: 2.5930827856063843
60th percentile: 2.667968940734863
70th percentile: 2.852389192581177
80th percentile: 2.9425382137298586
90th percentile: 10.927949213981629
95th percentile: 11.065190029144286
99th percentile: 11.249585628509521
mean time: 3.7817358414332074
Pipeline stage StressChecker completed in 117.24s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.77s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_78846_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_78846_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm50-combi_78846_v1 status is now torndown due to DeploymentManager action