chaiml-235b-sft-prod-rm_38783

developer_uid: rirv938
submission_id: chaiml-235b-sft-prod-rm_38783_v3
model_name: chaiml-235b-sft-prod-rm_38783_v3
model_group: ChaiML/235b_sft_prod_rm_
status: torndown
timestamp: 2026-02-22T01:21:39+00:00
num_battles: 12062
num_wins: 6579
celo_rating: 1327.75
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-235b-sft-prod-rm_38783_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-18
win_ratio: 0.5454319350024871
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_start|>', '###', '<|im_end|>', 'You:', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-prod-rm-38783-v3-uploader
Waiting for job on chaiml-235b-sft-prod-rm-38783-v3-uploader to finish
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
chaiml-235b-sft-prod-rm-38783-v3-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-38783-v3-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-38783-v3-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-38783-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-38783-v3-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16...
chaiml-235b-sft-prod-rm-38783-v3-uploader: Downloaded in 52.883s
chaiml-235b-sft-prod-rm-38783-v3-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 in 53.537s
chaiml-235b-sft-prod-rm-38783-v3-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-38783-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-38783-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-38783-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-38783-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-38783-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-38783-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-38783-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-38783-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/vocab.json
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/merges.txt
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/tokenizer.json
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v3-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00007-of-00027.safetensors?partNumber=33&uploadId=2~kkJmkQwOltSXgdR24G9f0AnM4twytDj": write tcp 10.0.33.234:34134->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-prod-rm-38783-v3-uploader: ERROR "cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00007-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~kkJmkQwOltSXgdR24G9f0AnM4twytDj caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000baea32065f4e964b-0069963cf7-156987ee57-default, host id:
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00006-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v3-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v3/default/model-00007-of-00027.safetensors
Job chaiml-235b-sft-prod-rm-38783-v3-uploader completed after 462.55s with status: succeeded
Stopping job with name chaiml-235b-sft-prod-rm-38783-v3-uploader
Pipeline stage VLLMUploader completed in 463.07s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-prod-rm-38783-v3
Waiting for inference service chaiml-235b-sft-prod-rm-38783-v3 to be ready
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-235b-sft-prod-rm-38783-v3 ready after 1176.0690710544586s
Pipeline stage VLLMDeployer completed in 1176.59s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.187978982925415s
Received healthy response to inference request in 2.040635108947754s
Received healthy response to inference request in 2.3572609424591064s
Received healthy response to inference request in 1.9393458366394043s
Received healthy response to inference request in 1.9426555633544922s
Received healthy response to inference request in 1.9137156009674072s
Received healthy response to inference request in 2.185164451599121s
Received healthy response to inference request in 2.1347620487213135s
Received healthy response to inference request in 2.401399850845337s
Received healthy response to inference request in 1.99574875831604s
Received healthy response to inference request in 2.139822244644165s
Received healthy response to inference request in 2.2241578102111816s
Received healthy response to inference request in 1.9679441452026367s
Received healthy response to inference request in 2.0522239208221436s
Received healthy response to inference request in 1.9550611972808838s
Received healthy response to inference request in 2.3198814392089844s
Received healthy response to inference request in 1.939621925354004s
Received healthy response to inference request in 2.0381603240966797s
Received healthy response to inference request in 2.0595924854278564s
Received healthy response to inference request in 2.3544020652770996s
Received healthy response to inference request in 2.2548556327819824s
Received healthy response to inference request in 2.003702402114868s
Received healthy response to inference request in 1.9456884860992432s
Received healthy response to inference request in 2.317394733428955s
Received healthy response to inference request in 2.040203332901001s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.2136380672454834s
Received healthy response to inference request in 1.9674835205078125s
Received healthy response to inference request in 2.0516605377197266s
Received healthy response to inference request in 2.1265599727630615s
Received healthy response to inference request in 2.395411968231201s
30 requests
0 failed requests
5th percentile: 1.939470076560974
10th percentile: 1.9423521995544433
20th percentile: 1.9649990558624268
30th percentile: 2.0013163089752197
40th percentile: 2.0404623985290526
50th percentile: 2.055908203125
60th percentile: 2.136786127090454
70th percentile: 2.1956767082214355
80th percentile: 2.2673634529113773
90th percentile: 2.3546879529953
95th percentile: 2.3782440066337585
99th percentile: 2.3996633648872376
mean time: 2.1155377785364786
Pipeline stage StressChecker completed in 66.38s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.68s
Shutdown handler de-registered
chaiml-235b-sft-prod-rm_38783_v3 status is now deployed due to DeploymentManager action
chaiml-235b-sft-prod-rm_38783_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-235b-sft-prod-rm_38783_v3 status is now torndown due to DeploymentManager action