developer_uid: acehao-chai
submission_id: chaiml-grpo-q235b-kimid_15958_v2
model_name: chaiml-grpo-q235b-kimid_15958_v2
model_group: ChaiML/grpo-q235b-kimid-
status: torndown
timestamp: 2026-02-24T07:46:42+00:00
num_battles: 10126
num_wins: 5424
celo_rating: 1321.94
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-q235b-kimid_15958_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: False
language_model: ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-17
win_ratio: 0.5356507999209954
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|im_end|>', '####', '<|user|>', '</think>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-q235b-kimid-15958-v2-uploader
Waiting for job on chaiml-grpo-q235b-kimid-15958-v2-uploader to finish
chaiml-grpo-q235b-kimid-15958-v2-uploader: Using quantization_mode: w4a16
chaiml-grpo-q235b-kimid-15958-v2-uploader: Checking if ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550-W4A16 already exists in ChaiML
chaiml-grpo-q235b-kimid-15958-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-q235b-kimid-15958-v2-uploader: Downloading snapshot of ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550-W4A16...
HTTP Request: %s %s "%s %d %s"
chaiml-grpo-q235b-kimid-15958-v2-uploader: Downloaded in 45.425s
chaiml-grpo-q235b-kimid-15958-v2-uploader: Processed model ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550 in 46.066s
chaiml-grpo-q235b-kimid-15958-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-q235b-kimid-15958-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-15958-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-q235b-kimid-15958-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-q235b-kimid-15958-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-q235b-kimid-15958-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-15958-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-15958-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-15958-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-15958-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-15958-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-15958-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-15958-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-15958-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-q235b-kimid-15958-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-q235b-kimid-15958-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-q235b-kimid-15958-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-q235b-kimid-15958-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-q235b-kimid-15958-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/.gitattributes
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/quantization_config.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/config.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/merges.txt
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/chat_template.jinja
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/special_tokens_map.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/tokenizer.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/added_tokens.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/vocab.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/generation_config.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/tokenizer_config.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model.safetensors.index.json
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00027-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00007-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00023-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00009-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00021-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00024-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00020-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00013-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00026-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00002-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00003-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00008-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00010-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00006-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00011-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00005-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00018-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00012-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00015-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00004-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00014-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00019-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00022-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00017-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v2/default/model-00001-of-00027.safetensors
Job chaiml-grpo-q235b-kimid-15958-v2-uploader completed after 308.78s with status: succeeded
Stopping job with name chaiml-grpo-q235b-kimid-15958-v2-uploader
Pipeline stage VLLMUploader completed in 309.26s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.80s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-q235b-kimid-15958-v2
Waiting for inference service chaiml-grpo-q235b-kimid-15958-v2 to be ready
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-grpo-q235b-kimid-15958-v2 ready after 844.6432526111603s
Pipeline stage VLLMDeployer completed in 845.19s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.160118818283081s
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Received healthy response to inference request in 2.1664693355560303s
Received healthy response to inference request in 1.879035234451294s
Received healthy response to inference request in 1.9205400943756104s
Received healthy response to inference request in 1.6962919235229492s
Received healthy response to inference request in 2.171677589416504s
Received healthy response to inference request in 2.231215238571167s
Received healthy response to inference request in 2.0915918350219727s
Received healthy response to inference request in 2.334531784057617s
Received healthy response to inference request in 2.253281831741333s
Received healthy response to inference request in 2.0912375450134277s
Received healthy response to inference request in 1.91041898727417s
Received healthy response to inference request in 2.000373601913452s
Received healthy response to inference request in 1.8472330570220947s
Received healthy response to inference request in 2.05006742477417s
Received healthy response to inference request in 2.029841899871826s
Received healthy response to inference request in 2.341881275177002s
Received healthy response to inference request in 2.3650288581848145s
Received healthy response to inference request in 1.9736037254333496s
Received healthy response to inference request in 2.2215030193328857s
Received healthy response to inference request in 2.25765323638916s
Received healthy response to inference request in 1.9772214889526367s
Received healthy response to inference request in 2.074514865875244s
Received healthy response to inference request in 2.0835626125335693s
Received healthy response to inference request in 2.0677502155303955s
Received healthy response to inference request in 2.364994764328003s
Received healthy response to inference request in 1.9399645328521729s
Received healthy response to inference request in 1.9419329166412354s
Received healthy response to inference request in 1.8878111839294434s
Received healthy response to inference request in 2.283616781234741s
30 requests
0 failed requests
5th percentile: 1.8615440368652343
10th percentile: 1.8869335889816283
20th percentile: 1.9360796451568603
30th percentile: 1.9761361598968505
40th percentile: 2.0419772148132322
50th percentile: 2.0790387392044067
60th percentile: 2.119002628326416
70th percentile: 2.186625218391418
80th percentile: 2.2541561126708984
90th percentile: 2.3352667331695556
95th percentile: 2.3545936942100525
99th percentile: 2.365018970966339
mean time: 2.0871655225753782
Pipeline stage StressChecker completed in 67.41s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.67s
Shutdown handler de-registered
chaiml-grpo-q235b-kimid_15958_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-q235b-kimid_15958_v2 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-grpo-q235b-kimid_15958_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-grpo-q235b-kimid-15958-v2 is running
Tearing down inference service chaiml-grpo-q235b-kimid-15958-v2
Service chaiml-grpo-q235b-kimid-15958-v2 has been torndown
Pipeline stage VLLMDeleter completed in 0.75s
run pipeline stage %s
HTTP Request: %s %s "%s %d %s"
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
clean up pipeline due to error=TeardownError("Got unexpected keyword argument 'request_checksum_calculation'")
Shutdown handler de-registered
chaiml-grpo-q235b-kimid_15958_v2 status is now torndown due to DeploymentManager action