chaiml-grpo-q235b-kimid_37540

developer_uid: acehao-chai
submission_id: chaiml-grpo-q235b-kimid_37540_v3
model_name: chaiml-grpo-q235b-kimid_37540_v3
model_group: ChaiML/grpo-q235b-kimid-
status: torndown
timestamp: 2026-03-11T17:37:51+00:00
num_battles: 589847
num_wins: 309735
celo_rating: 1323.34
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-q235b-kimid_37540_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: False
language_model: ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-25
win_ratio: 0.5251107490586542
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|im_end|>', '####', '<|user|>', '</think>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-q235b-kimid-37540-v3-uploader
Waiting for job on chaiml-grpo-q235b-kimid-37540-v3-uploader to finish
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-grpo-q235b-kimid-37540-v3-uploader: Using quantization_mode: w4a16
chaiml-grpo-q235b-kimid-37540-v3-uploader: Checking if ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300-W4A16 already exists in ChaiML
chaiml-grpo-q235b-kimid-37540-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-q235b-kimid-37540-v3-uploader: Downloading snapshot of ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300-W4A16...
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-grpo-q235b-kimid-37540-v3-uploader: Downloaded in 50.075s
chaiml-grpo-q235b-kimid-37540-v3-uploader: Processed model ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300 in 50.788s
chaiml-grpo-q235b-kimid-37540-v3-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-q235b-kimid-37540-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-q235b-kimid-37540-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-q235b-kimid-37540-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-q235b-kimid-37540-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-37540-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-37540-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-37540-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-37540-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-q235b-kimid-37540-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-q235b-kimid-37540-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-q235b-kimid-37540-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-q235b-kimid-37540-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-q235b-kimid-37540-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/generation_config.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/added_tokens.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/special_tokens_map.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/.gitattributes
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/quantization_config.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/chat_template.jinja
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/vocab.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/tokenizer.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model.safetensors.index.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/merges.txt
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/tokenizer_config.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/config.json
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00027-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00022-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00016-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00009-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00021-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00001-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00003-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00013-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00024-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00018-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00014-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00008-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00004-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00007-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00026-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00005-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00006-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00017-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00023-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00012-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00019-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00010-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00011-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00025-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00015-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00002-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v3-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v3/default/model-00020-of-00027.safetensors
Job chaiml-grpo-q235b-kimid-37540-v3-uploader completed after 135.58s with status: succeeded
Stopping job with name chaiml-grpo-q235b-kimid-37540-v3-uploader
Pipeline stage VLLMUploader completed in 136.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-q235b-kimid-37540-v3
Waiting for inference service chaiml-grpo-q235b-kimid-37540-v3 to be ready
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Retrying (%r) after connection broken by '%r': %s
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-grpo-q235b-kimid-37540-v3 ready after 453.32357573509216s
Pipeline stage VLLMDeployer completed in 453.88s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.009326457977295s
Received healthy response to inference request in 1.818460464477539s
Received healthy response to inference request in 2.220304489135742s
Received healthy response to inference request in 1.89534330368042s
Received healthy response to inference request in 2.0290627479553223s
Received healthy response to inference request in 2.2506344318389893s
Received healthy response to inference request in 2.1241164207458496s
Received healthy response to inference request in 1.9508156776428223s
Received healthy response to inference request in 1.9691393375396729s
Received healthy response to inference request in 2.0507357120513916s
Received healthy response to inference request in 1.9375834465026855s
Received healthy response to inference request in 2.2796430587768555s
Received healthy response to inference request in 2.08603572845459s
Received healthy response to inference request in 2.113483428955078s
Received healthy response to inference request in 1.840233564376831s
Received healthy response to inference request in 2.105804443359375s
Received healthy response to inference request in 1.9109022617340088s
Received healthy response to inference request in 1.99320650100708s
Received healthy response to inference request in 2.0402991771698s
Received healthy response to inference request in 2.0171799659729004s
Received healthy response to inference request in 2.2171008586883545s
Received healthy response to inference request in 1.9030787944793701s
Received healthy response to inference request in 1.8860702514648438s
Received healthy response to inference request in 2.137481212615967s
Received healthy response to inference request in 1.999392032623291s
Received healthy response to inference request in 2.135427951812744s
Received healthy response to inference request in 2.0354866981506348s
Received healthy response to inference request in 2.005786657333374s
Received healthy response to inference request in 1.9479057788848877s
Received healthy response to inference request in 2.2153658866882324s
30 requests
0 failed requests
5th percentile: 1.8608600735664367
10th percentile: 1.8944159984588622
20th percentile: 1.9322472095489502
30th percentile: 1.9636422395706177
40th percentile: 2.0032288074493407
50th percentile: 2.0231213569641113
60th percentile: 2.0444737911224364
70th percentile: 2.108108139038086
80th percentile: 2.1358386039733888
90th percentile: 2.217421221733093
95th percentile: 2.236985957622528
99th percentile: 2.2712305569648743
mean time: 2.037513558069865
Pipeline stage StressChecker completed in 65.63s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.64s
Shutdown handler de-registered
chaiml-grpo-q235b-kimid_37540_v3 status is now deployed due to DeploymentManager action
chaiml-grpo-q235b-kimid_37540_v3 status is now inactive due to system request
chaiml-grpo-q235b-kimid_37540_v3 status is now inactive due to Froze recruitment for AB test 0220_feynman
chaiml-grpo-q235b-kimid_37540_v3 status is now torndown due to DeploymentManager action