chaiml-grpo-glm50-combi_60857

developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-combi_60857_v2
model_name: chaiml-grpo-glm50-combi_60857_v2
model_group: ChaiML/grpo-glm50-combin
status: torndown
timestamp: 2026-04-04T00:51:50+00:00
num_battles: 11293
num_wins: 6682
celo_rating: 8406.88
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step300-int4-fix3
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_60857_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step300-int4-fix3
model_size: 31B
ranking_group: single
us_pacific_date: 2026-03-31
win_ratio: 0.5916939697157532
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '<|im_end|>', '####', '<|assistant|>', '</s>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-60857-v2-uploader
Waiting for job on chaiml-grpo-glm50-combi-60857-v2-uploader to finish
chaiml-grpo-glm50-combi-60857-v2-uploader: {repo_id} is already quantized
chaiml-grpo-glm50-combi-60857-v2-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-60857-v2-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step300-int4-fix3...
2026-03-31T22:29:41.180575+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:30:41.592333+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:31:41.713685+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-grpo-glm50-combi-60857-v2-uploader: Downloaded in 183.367s
2026-03-31T22:32:41.896912+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:33:42.123695+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-31T22:34:42.242955+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
chaiml-grpo-glm50-combi-60857-v2-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step300-int4-fix3 in 326.407s
chaiml-grpo-glm50-combi-60857-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-60857-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-60857-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-60857-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-60857-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-60857-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-60857-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-60857-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-60857-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-60857-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-60857-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-60857-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-60857-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-60857-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-60857-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-60857-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-60857-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-60857-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-60857-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/generation_config.json
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/chat_template.jinja
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/config.json
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/.gitattributes
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/quantization_config.json
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/tokenizer_config.json
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model.safetensors.index.json
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/tokenizer.json
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00075-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00075-of-00075.safetensors
2026-03-31T22:35:42.334266+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00047-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00047-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00052-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00052-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00014-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00014-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00032-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00032-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00064-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00064-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00062-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00062-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00043-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00043-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00004-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00004-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00008-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00008-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00031-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00031-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00039-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00039-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00024-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00024-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00065-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00065-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00035-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00035-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00049-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00049-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00048-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00048-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00037-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00037-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00033-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00033-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00041-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00041-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00016-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00016-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00050-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00050-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00040-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00040-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00028-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00028-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00051-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00051-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00007-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00007-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00006-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00006-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00058-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00058-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00046-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00046-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00002-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00002-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00013-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00013-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00070-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00070-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00012-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00012-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00044-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00044-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00053-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00053-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00059-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00059-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00011-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00011-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00029-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00029-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00021-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00021-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00034-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00034-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00060-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00060-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00045-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00045-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00054-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00054-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00023-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00023-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00068-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00068-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00025-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00025-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00019-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00019-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00042-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00042-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00001-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00001-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00020-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00020-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00022-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00022-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00030-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00030-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00063-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00063-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00005-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00005-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00069-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00069-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00074-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00074-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00071-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00071-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00038-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00038-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00010-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00010-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00026-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00026-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00057-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00057-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00027-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00027-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00036-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00036-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00072-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00072-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00061-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00061-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00073-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00073-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00009-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00009-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00017-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00017-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00067-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00067-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00066-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00066-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00055-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00055-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00015-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00015-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00018-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00018-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00056-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00056-of-00075.safetensors
chaiml-grpo-glm50-combi-60857-v2-uploader: cp /dev/shm/model_output/model-00003-of-00075.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-60857-v2/default/model-00003-of-00075.safetensors
Job chaiml-grpo-glm50-combi-60857-v2-uploader completed after 470.12s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-60857-v2-uploader
Pipeline stage VLLMUploader completed in 471.29s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.12s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.76s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-60857-v2
Waiting for inference service chaiml-grpo-glm50-combi-60857-v2 to be ready
2026-03-31T22:36:42.439795+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:37:42.542155+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:38:42.651756+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:39:42.760883+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
Failed to get response for submission chaiml-4d70-fd43-linear_51732_v5: ('http://guanaco-model-mesh-load-balancer-v2.model-mesh.kchai-google-us-east4.chaiverse.com/models/chaiml-4d70-fd43-linear_51732_v5/predict', '{"detail":"1 validation error for RuntimeResponse\\npredictions\\n  Field required [type=missing, input_value={\'detail\': \\"1 validation ...tic.dev/2.12/v/missing\\"}, input_type=dict]\\n    For further information visit https://errors.pydantic.dev/2.12/v/missing"}')
2026-03-31T22:40:42.860558+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:41:42.965632+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:42:43.067330+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:43:43.162256+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:44:43.263304+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:45:43.364468+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:46:43.470495+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:47:43.559413+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
2026-03-31T22:48:43.654595+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
Inference service chaiml-grpo-glm50-combi-60857-v2 ready after 772.0105590820312s
Pipeline stage VLLMDeployer completed in 772.46s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 11.049221515655518s
Received healthy response to inference request in 2.51291561126709s
2026-03-31T22:49:43.750803+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
Received healthy response to inference request in 2.510939598083496s
Received healthy response to inference request in 2.693413019180298s
Received healthy response to inference request in 2.7008564472198486s
Received healthy response to inference request in 2.778286933898926s
Received healthy response to inference request in 2.712614059448242s
Received healthy response to inference request in 2.6136436462402344s
Received healthy response to inference request in 2.807473659515381s
Received healthy response to inference request in 2.58939266204834s
Received healthy response to inference request in 2.663105010986328s
Received healthy response to inference request in 2.691204071044922s
Received healthy response to inference request in 2.737321615219116s
Received healthy response to inference request in 2.9671692848205566s
Received healthy response to inference request in 2.62717866897583s
Received healthy response to inference request in 2.6103501319885254s
Received healthy response to inference request in 3.086606979370117s
Received healthy response to inference request in 2.8392508029937744s
Received healthy response to inference request in 2.726707696914673s
Received healthy response to inference request in 2.578075885772705s
2026-03-31T22:50:43.849550+00:00 monitor updated for chaiml-grpo-glm50-combi_60857_v2
Received healthy response to inference request in 2.786158561706543s
Received healthy response to inference request in 2.7918591499328613s
Received healthy response to inference request in 2.4880378246307373s
Received healthy response to inference request in 2.547840118408203s
Received healthy response to inference request in 2.6464602947235107s
Received healthy response to inference request in 2.591184139251709s
Received healthy response to inference request in 2.5184617042541504s
Received healthy response to inference request in 2.580930233001709s
Received healthy response to inference request in 3.0450310707092285s
Received healthy response to inference request in 2.724696159362793s
30 requests
0 failed requests
5th percentile: 2.511828804016113
10th percentile: 2.5179070949554445
20th percentile: 2.580359363555908
30th percentile: 2.6046003341674804
40th percentile: 2.6387476444244387
50th percentile: 2.69230854511261
60th percentile: 2.7174468994140626
70th percentile: 2.749611210823059
80th percentile: 2.7949820518493653
90th percentile: 2.9749554634094237
95th percentile: 3.067897820472717
99th percentile: 8.740063300132759
mean time: 2.9738795518875123
Pipeline stage StressChecker completed in 101.32s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.29s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_60857_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_60857_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm50-combi_60857_v2 status is now torndown due to DeploymentManager action