chaiml-grpo-q235b-opusd_11130

developer_uid: acehao-chai
submission_id: chaiml-grpo-q235b-opusd_11130_v3
model_name: chaiml-grpo-q235b-opusd_11130_v3
model_group: ChaiML/grpo-q235b-opusd-
status: torndown
timestamp: 2026-03-11T17:33:26+00:00
num_battles: 475131
num_wins: 259736
celo_rating: 1329.51
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-q235b-opusd-v1-merged-chai-rm-step-700
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-q235b-opusd_11130_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: False
language_model: ChaiML/grpo-q235b-opusd-v1-merged-chai-rm-step-700
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-25
win_ratio: 0.5466618679900912
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|im_end|>', '####', '<|user|>', '</think>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-q235b-opusd-11130-v3-uploader
Waiting for job on chaiml-grpo-q235b-opusd-11130-v3-uploader to finish
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-grpo-q235b-opusd-11130-v3-uploader: Using quantization_mode: w4a16
chaiml-grpo-q235b-opusd-11130-v3-uploader: Checking if ChaiML/grpo-q235b-opusd-v1-merged-chai-rm-step-700-W4A16 already exists in ChaiML
chaiml-grpo-q235b-opusd-11130-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-q235b-opusd-11130-v3-uploader: Downloading snapshot of ChaiML/grpo-q235b-opusd-v1-merged-chai-rm-step-700-W4A16...
chaiml-grpo-q235b-opusd-11130-v3-uploader: Downloaded in 50.338s
chaiml-grpo-q235b-opusd-11130-v3-uploader: Processed model ChaiML/grpo-q235b-opusd-v1-merged-chai-rm-step-700 in 50.990s
chaiml-grpo-q235b-opusd-11130-v3-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-q235b-opusd-11130-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-opusd-11130-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-q235b-opusd-11130-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-q235b-opusd-11130-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-q235b-opusd-11130-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-opusd-11130-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-q235b-opusd-11130-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-opusd-11130-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-q235b-opusd-11130-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-opusd-11130-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-q235b-opusd-11130-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-opusd-11130-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-q235b-opusd-11130-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-q235b-opusd-11130-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-q235b-opusd-11130-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-q235b-opusd-11130-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-q235b-opusd-11130-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-q235b-opusd-11130-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/.gitattributes
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/added_tokens.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/config.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/special_tokens_map.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/generation_config.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/quantization_config.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/chat_template.jinja
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/merges.txt
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/tokenizer_config.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/vocab.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/tokenizer.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model.safetensors.index.json
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00027-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00004-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00021-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00014-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00008-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00023-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00006-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00017-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00005-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00002-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00019-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00003-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00001-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00012-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00009-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00010-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00011-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00026-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00018-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00013-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00025-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00016-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00015-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00024-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00020-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00007-of-00027.safetensors
chaiml-grpo-q235b-opusd-11130-v3-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-opusd-11130-v3/default/model-00022-of-00027.safetensors
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Job chaiml-grpo-q235b-opusd-11130-v3-uploader completed after 136.25s with status: succeeded
Stopping job with name chaiml-grpo-q235b-opusd-11130-v3-uploader
Pipeline stage VLLMUploader completed in 139.99s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.10s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-q235b-opusd-11130-v3
Waiting for inference service chaiml-grpo-q235b-opusd-11130-v3 to be ready
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-grpo-q235b-opusd-11130-v3 ready after 381.8528792858124s
Pipeline stage VLLMDeployer completed in 389.28s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.522914171218872s
Received healthy response to inference request in 3.654745578765869s
Received healthy response to inference request in 1.9998841285705566s
Received healthy response to inference request in 2.8743977546691895s
Received healthy response to inference request in 3.259796142578125s
Received healthy response to inference request in 2.12505841255188s
Received healthy response to inference request in 1.8632359504699707s
Received healthy response to inference request in 2.3847861289978027s
Received healthy response to inference request in 2.0950028896331787s
Received healthy response to inference request in 1.9385414123535156s
Received healthy response to inference request in 1.89121413230896s
Received healthy response to inference request in 1.9076509475708008s
Received healthy response to inference request in 1.8851637840270996s
Received healthy response to inference request in 1.7897236347198486s
Received healthy response to inference request in 2.0376198291778564s
Received healthy response to inference request in 1.790698766708374s
Received healthy response to inference request in 1.9624571800231934s
Received healthy response to inference request in 2.109912157058716s
Received healthy response to inference request in 2.207669258117676s
Received healthy response to inference request in 2.1300361156463623s
Received healthy response to inference request in 1.9026942253112793s
Received healthy response to inference request in 1.8999676704406738s
Received healthy response to inference request in 1.920135259628296s
Received healthy response to inference request in 2.0530855655670166s
Received healthy response to inference request in 2.1127822399139404s
Received healthy response to inference request in 2.0951404571533203s
Received healthy response to inference request in 2.164217233657837s
Received healthy response to inference request in 1.734682321548462s
Received healthy response to inference request in 1.9922006130218506s
Received healthy response to inference request in 1.9729971885681152s
30 requests
0 failed requests
5th percentile: 1.790162444114685
10th percentile: 1.855982232093811
20th percentile: 1.898216962814331
30th percentile: 1.9163899660110473
40th percentile: 1.9687811851501464
50th percentile: 2.0187519788742065
60th percentile: 2.0950579166412355
70th percentile: 2.1164650917053223
80th percentile: 2.172907638549805
90th percentile: 2.9129375934600836
95th percentile: 3.4045110583305354
99th percentile: 3.6165144705772403
mean time: 2.1759470383326214
Pipeline stage StressChecker completed in 80.62s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.65s
Shutdown handler de-registered
chaiml-grpo-q235b-opusd_11130_v3 status is now deployed due to DeploymentManager action
chaiml-grpo-q235b-opusd_11130_v3 status is now inactive due to system request
chaiml-grpo-q235b-opusd_11130_v3 status is now inactive due to Froze recruitment for AB test 0220_feynman
chaiml-grpo-q235b-kimid_83709_v2 status is now torndown due to DeploymentManager action
Deleting key chaiml-grpo-q235b-opusd-34252-v2/default/vocab.json from bucket guanaco-vllm-models
chaiml-grpo-q235b-opusd_11130_v3 status is now torndown due to DeploymentManager action