chaiml-sft-qwen-235b-ro_79255

developer_uid: rirv938
submission_id: chaiml-sft-qwen-235b-ro_79255_v1
model_name: chaiml-sft-qwen-235b-ro_79255_v1
model_group: ChaiML/sft_qwen_235b_rob
status: torndown
timestamp: 2026-02-23T04:41:59+00:00
num_battles: 10316
num_wins: 5801
celo_rating: 1339.25
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/sft_qwen_235b_rob_new_2_version-W4A16
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 96
reward_model: default
display_name: chaiml-sft-qwen-235b-ro_79255_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/sft_qwen_235b_rob_new_2_version-W4A16
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-19
win_ratio: 0.5623303606048856
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '###', '</s>', '<|im_start|>', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 96}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-sft-qwen-235b-ro-79255-v1-uploader
Waiting for job on chaiml-sft-qwen-235b-ro-79255-v1-uploader to finish
chaiml-sft-qwen-235b-ro-79255-v1-uploader: Using quantization_mode: w4a16
chaiml-sft-qwen-235b-ro-79255-v1-uploader: Repo ChaiML/sft_qwen_235b_rob_new_2_version-W4A16 already ends in W4A16. Skipping...
chaiml-sft-qwen-235b-ro-79255-v1-uploader: Checking if ChaiML/sft_qwen_235b_rob_new_2_version-W4A16 already exists in ChaiML
chaiml-sft-qwen-235b-ro-79255-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-sft-qwen-235b-ro-79255-v1-uploader: Downloading snapshot of ChaiML/sft_qwen_235b_rob_new_2_version-W4A16...
chaiml-sft-qwen-235b-ro-79255-v1-uploader: Downloaded in 67.835s
chaiml-sft-qwen-235b-ro-79255-v1-uploader: Processed model ChaiML/sft_qwen_235b_rob_new_2_version-W4A16 in 68.415s
chaiml-sft-qwen-235b-ro-79255-v1-uploader: creating bucket guanaco-vllm-models
chaiml-sft-qwen-235b-ro-79255-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-sft-qwen-235b-ro-79255-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-sft-qwen-235b-ro-79255-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-sft-qwen-235b-ro-79255-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-sft-qwen-235b-ro-79255-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-sft-qwen-235b-ro-79255-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-sft-qwen-235b-ro-79255-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-sft-qwen-235b-ro-79255-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-sft-qwen-235b-ro-79255-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-sft-qwen-235b-ro-79255-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-sft-qwen-235b-ro-79255-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-sft-qwen-235b-ro-79255-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-sft-qwen-235b-ro-79255-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-sft-qwen-235b-ro-79255-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-sft-qwen-235b-ro-79255-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-sft-qwen-235b-ro-79255-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-sft-qwen-235b-ro-79255-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-sft-qwen-235b-ro-79255-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/added_tokens.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/quantization_config.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/tokenizer_config.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/special_tokens_map.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/chat_template.jinja
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/.gitattributes
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/generation_config.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/merges.txt
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/config.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/vocab.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/tokenizer.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model.safetensors.index.json
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00027-of-00027.safetensors
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00016-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00013-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00020-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00005-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00009-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00002-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00017-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00010-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00008-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00012-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00018-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00004-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00001-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00026-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00015-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00014-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00007-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00019-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00006-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00023-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00011-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00003-of-00027.safetensors
chaiml-sft-qwen-235b-ro-79255-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-79255-v1/default/model-00025-of-00027.safetensors
Job chaiml-sft-qwen-235b-ro-79255-v1-uploader completed after 257.01s with status: succeeded
Stopping job with name chaiml-sft-qwen-235b-ro-79255-v1-uploader
Pipeline stage VLLMUploader completed in 257.54s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.02s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-sft-qwen-235b-ro-79255-v1
Waiting for inference service chaiml-sft-qwen-235b-ro-79255-v1 to be ready
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-sft-qwen-235b-ro-79255-v1 ready after 1144.8780658245087s
Pipeline stage VLLMDeployer completed in 1145.53s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.4723594188690186s
Received healthy response to inference request in 2.3273279666900635s
Received healthy response to inference request in 2.2707419395446777s
Received healthy response to inference request in 2.3509573936462402s
Received healthy response to inference request in 2.323087692260742s
Received healthy response to inference request in 2.3651440143585205s
Received healthy response to inference request in 2.260021686553955s
Received healthy response to inference request in 2.3551247119903564s
Received healthy response to inference request in 2.3463428020477295s
Received healthy response to inference request in 2.499063014984131s
Received healthy response to inference request in 2.2939488887786865s
Received healthy response to inference request in 2.538949728012085s
Received healthy response to inference request in 2.339890718460083s
Received healthy response to inference request in 2.4506843090057373s
Received healthy response to inference request in 2.7731027603149414s
Received healthy response to inference request in 2.5594911575317383s
Received healthy response to inference request in 2.4016671180725098s
Received healthy response to inference request in 2.4635894298553467s
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Received healthy response to inference request in 2.5442941188812256s
Received healthy response to inference request in 2.268258810043335s
Received healthy response to inference request in 2.292954683303833s
Received healthy response to inference request in 2.500959634780884s
Received healthy response to inference request in 2.403850555419922s
Received healthy response to inference request in 2.149920701980591s
Received healthy response to inference request in 2.3822195529937744s
Received healthy response to inference request in 2.2787981033325195s
Received healthy response to inference request in 2.320624589920044s
Received healthy response to inference request in 2.2996397018432617s
Received healthy response to inference request in 2.459665536880493s
Received healthy response to inference request in 2.3610689640045166s
30 requests
0 failed requests
5th percentile: 2.263728392124176
10th percentile: 2.2704936265945435
20th percentile: 2.293750047683716
30th percentile: 2.3223487615585325
40th percentile: 2.3437619686126707
50th percentile: 2.3580968379974365
60th percentile: 2.3899985790252685
70th percentile: 2.453378677368164
80th percentile: 2.477700138092041
90th percentile: 2.539484167098999
95th percentile: 2.5526524901390077
99th percentile: 2.7111553955078125
mean time: 2.388458323478699
Pipeline stage StressChecker completed in 77.40s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.72s
Shutdown handler de-registered
chaiml-sft-qwen-235b-ro_79255_v1 status is now deployed due to DeploymentManager action
chaiml-sft-qwen-235b-ro_79255_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-sft-qwen-235b-ro_79255_v1 status is now torndown due to DeploymentManager action