chaiml-grpo-glm50-combi_46874

developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-combi_46874_v4
model_name: chaiml-grpo-glm50-combi_46874_v4
model_group: ChaiML/grpo-glm50-combin
status: inactive
timestamp: 2026-03-26T03:55:04+00:00
num_battles: 19135
num_wins: 11117
celo_rating: 1351.55
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_46874_v4
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast
model_size: 31B
ranking_group: single
us_pacific_date: 2026-03-25
win_ratio: 0.5809772667886073
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|assistant|>', '</s>', '<|im_end|>', '<|user|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-46874-v4-uploader
Waiting for job on chaiml-grpo-glm50-combi-46874-v4-uploader to finish
chaiml-grpo-glm50-combi-46874-v4-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-46874-v4-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast...
2026-03-25T20:23:31.942872+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
Failed to get response for submission blend_kases_2026-03-25: HTTPConnectionPool(host='chaiml-kimid-v12-mv1-to-30309-v2-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
2026-03-25T20:24:32.028226+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
chaiml-grpo-glm50-combi-46874-v4-uploader: Downloaded in 104.072s
2026-03-25T20:25:32.114837+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-25T20:26:32.212590+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
chaiml-grpo-glm50-combi-46874-v4-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast in 249.312s
chaiml-grpo-glm50-combi-46874-v4-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-46874-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-46874-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-46874-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-46874-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-46874-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-46874-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-46874-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-46874-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-46874-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-46874-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-46874-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-46874-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-grpo-glm50-combi-46874-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/chat_template.jinja
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/config.json
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/.gitattributes
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/generation_config.json
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/quantization_config.json
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/tokenizer_config.json
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model.safetensors.index.json
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/tokenizer.json
2026-03-25T20:27:32.328555+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
Retrying (%r) after connection broken by '%r': %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Retrying (%r) after connection broken by '%r': %s
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00077-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00077-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00078-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00078-of-00078.safetensors
2026-03-25T20:28:32.432137+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00003-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00003-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00036-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00036-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00008-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00008-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00007-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00007-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00005-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00005-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00019-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00019-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00024-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00024-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00002-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00002-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00014-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00014-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00032-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00032-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00018-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00018-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00006-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00006-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00009-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00009-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00022-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00022-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00056-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00056-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00010-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00010-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00034-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00034-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00047-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00047-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00065-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00065-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00053-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00053-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00004-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00004-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00001-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00001-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00029-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00029-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00043-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00043-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00015-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00015-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00045-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00045-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00023-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00023-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00066-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00066-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00017-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00017-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00040-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00040-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00059-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00059-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00026-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00026-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00044-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00044-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00021-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00021-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00041-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00041-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00030-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00030-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00035-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00035-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00039-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00039-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00055-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00055-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00042-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00042-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00031-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00031-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00033-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00033-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00020-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00020-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00060-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00060-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00073-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00073-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00069-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00069-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00025-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00025-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00038-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00038-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00013-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00013-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00037-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00037-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00067-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00067-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00016-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00016-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00046-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00046-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00051-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00051-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00048-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00048-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00027-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00027-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00072-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00072-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00062-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00062-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00075-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00075-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00052-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00052-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00057-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00057-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00012-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00012-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00050-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00050-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00068-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00068-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00028-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00028-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00070-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00070-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00064-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00064-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00076-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00076-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00049-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00049-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00058-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00058-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00063-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00063-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00071-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00071-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00074-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00074-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00054-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00054-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v4-uploader: cp /dev/shm/model_output/model-00061-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v4/default/model-00061-of-00078.safetensors
Job chaiml-grpo-glm50-combi-46874-v4-uploader completed after 401.89s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-46874-v4-uploader
Pipeline stage VLLMUploader completed in 402.42s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.46s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-46874-v4
Waiting for inference service chaiml-grpo-glm50-combi-46874-v4 to be ready
2026-03-25T20:29:32.526926+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
2026-03-25T20:30:32.675279+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-25T20:31:32.796846+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
2026-03-25T20:32:32.922792+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
2026-03-25T20:33:33.322715+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
Failed to get request counts for guanaco-submitter. Falling back to default
Failed to get response for submission qwen-qwen3-5-35b-a3b_v53: ('http://qwen-qwen3-5-35b-a3b-v53-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'request timeout')
2026-03-25T20:34:33.449863+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
2026-03-25T20:35:33.557831+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service chaiml-grpo-glm50-combi-46874-v4 ready after 400.9375500679016s
Pipeline stage VLLMDeployer completed in 401.44s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 10.86023211479187s
Received healthy response to inference request in 2.4607133865356445s
Received healthy response to inference request in 2.4955687522888184s
Received healthy response to inference request in 2.5543324947357178s
Received healthy response to inference request in 2.5284698009490967s
Received healthy response to inference request in 2.649293899536133s
2026-03-25T20:36:33.678697+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
Received healthy response to inference request in 11.164669275283813s
Received healthy response to inference request in 2.6321587562561035s
Received healthy response to inference request in 3.0782485008239746s
Received healthy response to inference request in 2.566610336303711s
Received healthy response to inference request in 2.5775272846221924s
Received healthy response to inference request in 12.187369346618652s
Received healthy response to inference request in 2.762113332748413s
Received healthy response to inference request in 2.8389947414398193s
Received healthy response to inference request in 2.992673397064209s
Received healthy response to inference request in 2.6592955589294434s
Received healthy response to inference request in 2.9452402591705322s
Received healthy response to inference request in 2.8173742294311523s
Received healthy response to inference request in 2.6269819736480713s
Received healthy response to inference request in 2.5724527835845947s
Received healthy response to inference request in 2.5858232975006104s
Received healthy response to inference request in 2.5084052085876465s
Received healthy response to inference request in 2.6847894191741943s
Received healthy response to inference request in 2.8582651615142822s
2026-03-25T20:37:33.795371+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v4
Received healthy response to inference request in 2.6669747829437256s
Received healthy response to inference request in 2.773486614227295s
Received healthy response to inference request in 2.614229202270508s
Received healthy response to inference request in 3.0294573307037354s
Received healthy response to inference request in 2.6313862800598145s
Received healthy response to inference request in 2.798544406890869s
30 requests
0 failed requests
5th percentile: 2.501345157623291
10th percentile: 2.526463341712952
20th percentile: 2.571284294128418
30th percentile: 2.6057074308395385
40th percentile: 2.631849765777588
50th percentile: 2.6631351709365845
60th percentile: 2.7666626453399656
70th percentile: 2.8238603830337525
80th percentile: 2.9547268867492678
90th percentile: 3.8564468622207753
95th percentile: 11.027672553062438
99th percentile: 11.89078632593155
mean time: 3.570722730954488
Pipeline stage StressChecker completed in 111.08s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.83s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_46874_v4 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_46874_v4 status is now inactive due to system request