chaiml-grpo-glm50-combi_46874

developer_uid: richhx
submission_id: chaiml-grpo-glm50-combi_46874_v3
model_name: chaiml-grpo-glm50-combi_46874_v3
model_group: ChaiML/grpo-glm50-combin
status: torndown
timestamp: 2026-03-26T17:15:42+00:00
num_battles: 10660
num_wins: 6114
celo_rating: 1344.41
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_46874_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast
model_size: 31B
ranking_group: single
us_pacific_date: 2026-03-25
win_ratio: 0.5735459662288931
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_start|>', '<|im_end|>', '</s>', '###', 'You:'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-46874-v3-uploader
Waiting for job on chaiml-grpo-glm50-combi-46874-v3-uploader to finish
2026-03-25T20:19:02.482052+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
chaiml-grpo-glm50-combi-46874-v3-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-46874-v3-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast...
2026-03-25T20:20:02.575960+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
2026-03-25T20:21:02.671867+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
chaiml-grpo-glm50-combi-46874-v3-uploader: Downloaded in 113.223s
2026-03-25T20:22:02.768674+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
2026-03-25T20:23:02.958616+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
%s, retrying in %s seconds...
%s, retrying in %s seconds...
2026-03-25T20:24:03.065548+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Connection pool is full, discarding connection: %s. Connection pool size: %s
%s, retrying in %s seconds...
2026-03-25T20:25:03.155720+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
%s, retrying in %s seconds...
2026-03-25T20:26:03.244787+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Job failed chaiml-grpo-glm50-combi-46874-v3-uploader: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 25 Mar 2026 20:26:50 GMT', 'Content-Type': 'application/json', 'Content-Length': '272', 'Connection': 'keep-alive', 'audit-id': 'cd1e69a7-a385-4a17-9862-7fe4d97d4008', 'Cache-Control': 'no-cache, private', 'x-kubernetes-pf-flowschema-uid': '757159af-7462-4610-a5a7-751d984daa90', 'x-kubernetes-pf-prioritylevel-uid': '5f6bf145-f133-455b-a330-c9bf3e5ea771', 'cf-cache-status': 'DYNAMIC', 'set-cookie': '__cf_bm=sCdunYW6.0noVaXPOri_NoHwMeDD75dmOG4hohnjY4s-1774470410.7792494-1.0.1.1-ITPkYWAug3UtsPcQNdAKac1bLrkQqLfUVy72abnrJv9P6gKKJpWMnZQRcZOKNONQMZ.OYMLgZjFMUcVPHkB8V; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Wed, 25 Mar 2026 20:56:50 GMT', 'Server': 'cloudflare', 'CF-RAY': '9e20a7235bfee5e9-IAD'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"chaiml-grpo-glm50-combi-46874-v3-uploader\" not found","reason":"NotFound","details":{"name":"chaiml-grpo-glm50-combi-46874-v3-uploader","group":"batch","kind":"jobs"},"code":404}

Stopping job with name chaiml-grpo-glm50-combi-46874-v3-uploader
%s, retrying in %s seconds...
Stopping job with name chaiml-grpo-glm50-combi-46874-v3-uploader
%s, retrying in %s seconds...
Stopping job with name chaiml-grpo-glm50-combi-46874-v3-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-grpo-glm50-combi-46874-v3-uploader
Waiting for job on chaiml-grpo-glm50-combi-46874-v3-uploader to finish
2026-03-25T20:27:03.337034+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
chaiml-grpo-glm50-combi-46874-v3-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-46874-v3-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast...
2026-03-25T20:28:03.463469+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-25T20:29:03.583206+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
chaiml-grpo-glm50-combi-46874-v3-uploader: Downloaded in 118.199s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-25T20:30:03.714419+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-25T20:31:03.841396+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Failed to get request counts for guanaco-submitter. Falling back to default
2026-03-25T20:32:03.942979+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
2026-03-25T20:33:04.040700+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
2026-03-25T20:34:04.143077+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
chaiml-grpo-glm50-combi-46874-v3-uploader: Processed model ChaiML/grpo-glm50-combine-rm721-mega-kimi-mix8020-step100-int4-fast in 411.704s
chaiml-grpo-glm50-combi-46874-v3-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-46874-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-46874-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-46874-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-46874-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-46874-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-46874-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-46874-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-46874-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-46874-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-46874-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-46874-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-46874-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-46874-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-46874-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/config.json
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/generation_config.json
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/.gitattributes
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/chat_template.jinja
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/tokenizer_config.json
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/quantization_config.json
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model.safetensors.index.json
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/tokenizer.json
2026-03-25T20:35:04.252295+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00077-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00077-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00078-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00078-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00032-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00032-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00005-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00005-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00006-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00006-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00003-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00003-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00014-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00014-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00048-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00048-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00016-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00016-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00007-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00007-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00033-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00033-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00047-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00047-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00017-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00017-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00037-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00037-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00022-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00022-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00038-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00038-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00019-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00019-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00001-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00001-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00021-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00021-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00009-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00009-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00034-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00034-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00046-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00046-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00049-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00049-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00011-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00011-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00010-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00010-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00029-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00029-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00042-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00042-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00002-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00002-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00023-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00023-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00018-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00018-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00028-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00028-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00050-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00050-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00013-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00013-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00008-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00008-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00030-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00030-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00057-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00057-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00070-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00070-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00027-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00027-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00072-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00072-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00015-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00015-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00076-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00076-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00031-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00031-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00056-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00056-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00059-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00059-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00026-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00026-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00074-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00074-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00004-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00004-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00045-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00045-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00060-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00060-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00067-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00067-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00064-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00064-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00044-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00044-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00041-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00041-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00035-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00035-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00012-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00012-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00066-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00066-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00040-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00040-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00069-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00069-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00043-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00043-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00039-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00039-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00062-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00062-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00051-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00051-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00058-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00058-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00036-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00036-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00020-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00020-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00065-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00065-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00024-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00024-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00073-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00073-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00061-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00061-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00068-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00068-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00071-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00071-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00054-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00054-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00063-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00063-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00055-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00055-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00052-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00052-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00075-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00075-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00025-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00025-of-00078.safetensors
chaiml-grpo-glm50-combi-46874-v3-uploader: cp /dev/shm/model_output/model-00053-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-46874-v3/default/model-00053-of-00078.safetensors
2026-03-25T20:36:04.360379+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Job chaiml-grpo-glm50-combi-46874-v3-uploader completed after 564.73s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-46874-v3-uploader
Pipeline stage VLLMUploader completed in 1093.52s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.90s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-46874-v3
Waiting for inference service chaiml-grpo-glm50-combi-46874-v3 to be ready
2026-03-25T20:37:04.473041+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
2026-03-25T20:38:04.644157+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Failed to get response for submission chaiml-reward-fresh-dat_37656_v2: ('http://guanaco-model-mesh-load-balancer-v2.model-mesh.kchai-google-us-east4.chaiverse.com/models/chaiml-4d70-fd43-linear_51732_v5/predict', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination')
2026-03-25T20:39:04.770753+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
2026-03-25T20:40:04.892496+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
2026-03-25T20:41:05.017160+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
2026-03-25T20:42:05.166373+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Inference service chaiml-grpo-glm50-combi-46874-v3 ready after 391.2135879993439s
Pipeline stage VLLMDeployer completed in 391.85s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 11.225099086761475s
2026-03-25T20:43:05.744488+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Received healthy response to inference request in 2.6647965908050537s
Received healthy response to inference request in 2.8950307369232178s
Received healthy response to inference request in 2.768113374710083s
Received healthy response to inference request in 2.7585031986236572s
Failed to get response for submission qwen-qwen3-5-35b-a3b-fp8_v11: ('http://qwen-qwen3-5-35b-a3b-fp8-v11-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'request timeout')
Received healthy response to inference request in 2.8589203357696533s
Received healthy response to inference request in 2.8230605125427246s
Received healthy response to inference request in 2.664151906967163s
Received healthy response to inference request in 2.7304935455322266s
Received healthy response to inference request in 2.829850196838379s
Received healthy response to inference request in 3.033167839050293s
Received healthy response to inference request in 2.766572952270508s
Received healthy response to inference request in 2.5222949981689453s
Received healthy response to inference request in 2.512129545211792s
Received healthy response to inference request in 2.7746481895446777s
Received healthy response to inference request in 2.6243181228637695s
Received healthy response to inference request in 2.6485490798950195s
Received healthy response to inference request in 2.7178618907928467s
Received healthy response to inference request in 2.720698118209839s
Received healthy response to inference request in 2.911346435546875s
Received healthy response to inference request in 3.0096375942230225s
Received healthy response to inference request in 2.7322888374328613s
2026-03-25T20:44:05.872892+00:00 monitor updated for chaiml-grpo-glm50-combi_46874_v3
Received healthy response to inference request in 2.68198299407959s
Received healthy response to inference request in 2.629913330078125s
Received healthy response to inference request in 2.607210397720337s
Received healthy response to inference request in 2.675114631652832s
Received healthy response to inference request in 2.593801736831665s
Received healthy response to inference request in 2.942551851272583s
Received healthy response to inference request in 2.6523356437683105s
Received healthy response to inference request in 2.6160688400268555s
30 requests
0 failed requests
5th percentile: 2.554473030567169
10th percentile: 2.6058695316314697
20th percentile: 2.628794288635254
30th percentile: 2.660607028007507
40th percentile: 2.679235649108887
50th percentile: 2.7255958318710327
60th percentile: 2.7617311000823976
70th percentile: 2.7891718864440915
80th percentile: 2.866142416000366
90th percentile: 2.949260425567627
95th percentile: 3.0225792288780213
99th percentile: 8.849439024925239
mean time: 3.019683750470479
Pipeline stage StressChecker completed in 94.77s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.58s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_46874_v3 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_46874_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-glm50-combi_46874_v3 status is now torndown due to DeploymentManager action