developer_uid: chai_backend_admin
submission_id: chaiml-merged-qwen-35-d_39140_v4
model_name: chaiml-merged-qwen-35-d_39140_v4
model_group: ChaiML/merged_qwen_35_dp
status: torndown
timestamp: 2026-03-29T02:57:09+00:00
num_battles: 10800
num_wins: 5751
celo_rating: 1318.05
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/merged_qwen_35_dpo_lower_lr_v
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-merged-qwen-35-d_39140_v4
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/merged_qwen_35_dpo_lower_lr_v
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-25
win_ratio: 0.5325
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_start|>', '<|im_end|>', '###', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<think></think><|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-merged-qwen-35-d-39140-v4-uploader
Waiting for job on chaiml-merged-qwen-35-d-39140-v4-uploader to finish
chaiml-merged-qwen-35-d-39140-v4-uploader: Using quantization_mode: fp8
chaiml-merged-qwen-35-d-39140-v4-uploader: Checking if ChaiML/merged_qwen_35_dpo_lower_lr_v-FP8 already exists in ChaiML
chaiml-merged-qwen-35-d-39140-v4-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-merged-qwen-35-d-39140-v4-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v-FP8...
2026-03-25T23:17:49.754663+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
chaiml-merged-qwen-35-d-39140-v4-uploader: Downloaded in 31.819s
chaiml-merged-qwen-35-d-39140-v4-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 34.282s
chaiml-merged-qwen-35-d-39140-v4-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-d-39140-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-d-39140-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-d-39140-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-d-39140-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-d-39140-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-d-39140-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-d-39140-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-d-39140-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-d-39140-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-d-39140-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-d-39140-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-d-39140-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-merged-qwen-35-d-39140-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default
chaiml-merged-qwen-35-d-39140-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default/generation_config.json
chaiml-merged-qwen-35-d-39140-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default/config.json
chaiml-merged-qwen-35-d-39140-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default/tokenizer_config.json
chaiml-merged-qwen-35-d-39140-v4-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default/recipe.yaml
chaiml-merged-qwen-35-d-39140-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default/chat_template.jinja
chaiml-merged-qwen-35-d-39140-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default/.gitattributes
chaiml-merged-qwen-35-d-39140-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default/tokenizer.json
2026-03-25T23:18:50.045431+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
chaiml-merged-qwen-35-d-39140-v4-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v4/default/model.safetensors
2026-03-25T23:19:50.254967+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Job chaiml-merged-qwen-35-d-39140-v4-uploader completed after 180.35s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-d-39140-v4-uploader
Pipeline stage VLLMUploader completed in 181.56s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.02s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-merged-qwen-35-d-39140-v4
Waiting for inference service chaiml-merged-qwen-35-d-39140-v4 to be ready
2026-03-25T23:20:50.428398+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
2026-03-25T23:21:50.619114+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Inference service chaiml-merged-qwen-35-d-39140-v4 ready after 172.1853485107422s
Pipeline stage VLLMDeployer completed in 173.10s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-25T23:22:50.793895+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received unhealthy response to inference request!
HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received unhealthy response to inference request!
HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received unhealthy response to inference request!
2026-03-25T23:23:50.969940+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received unhealthy response to inference request!
HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received unhealthy response to inference request!
Received healthy response to inference request in 18.50430989265442s
2026-03-25T23:24:51.159336+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Received healthy response to inference request in 10.970229148864746s
HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received unhealthy response to inference request!
HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received unhealthy response to inference request!
2026-03-25T23:25:51.337725+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Received healthy response to inference request in 10.778024911880493s
HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received unhealthy response to inference request!
Received healthy response to inference request in 5.865659952163696s
Received healthy response to inference request in 5.950884580612183s
Received healthy response to inference request in 5.751771688461304s
Received healthy response to inference request in 10.867347955703735s
2026-03-25T23:26:51.517729+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Received healthy response to inference request in 5.568403005599976s
Received healthy response to inference request in 5.7257399559021s
Received healthy response to inference request in 5.4070143699646s
Received healthy response to inference request in 5.413041353225708s
Received healthy response to inference request in 5.69402813911438s
Received healthy response to inference request in 5.602294921875s
Received healthy response to inference request in 5.857217311859131s
Received healthy response to inference request in 5.568380832672119s
Received healthy response to inference request in 5.437145948410034s
Received healthy response to inference request in 5.514057397842407s
Received healthy response to inference request in 5.39903712272644s
2026-03-25T23:27:51.690195+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Received healthy response to inference request in 5.773033380508423s
Received healthy response to inference request in 5.679366588592529s
Received healthy response to inference request in 5.468403100967407s
Received healthy response to inference request in 5.731119394302368s
30 requests
8 failed requests
5th percentile: 5.409726512432099
10th percentile: 5.434735488891602
20th percentile: 5.557516145706177
30th percentile: 5.65624508857727
40th percentile: 5.728967618942261
50th percentile: 5.815125346183777
60th percentile: 7.8817407131195
70th percentile: 13.230453372001627
80th percentile: 20.406040287017824
90th percentile: 20.426072740554808
95th percentile: 20.623038709163666
99th percentile: 26.17364594221116
mean time: 10.805400180816651
%s, retrying in %s seconds...
Received healthy response to inference request in 5.737289905548096s
Received healthy response to inference request in 5.637301921844482s
Received healthy response to inference request in 5.537476539611816s
Received healthy response to inference request in 5.374013185501099s
Received healthy response to inference request in 5.485329866409302s
Received healthy response to inference request in 5.911117076873779s
2026-03-25T23:28:51.955170+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Received healthy response to inference request in 10.873599767684937s
Received healthy response to inference request in 5.497524738311768s
Received healthy response to inference request in 5.477458953857422s
Received healthy response to inference request in 5.553410291671753s
Received healthy response to inference request in 5.618671417236328s
Received healthy response to inference request in 5.407721996307373s
Received healthy response to inference request in 5.654375791549683s
Received healthy response to inference request in 5.713266134262085s
Received healthy response to inference request in 5.553391933441162s
Received healthy response to inference request in 5.64301872253418s
2026-03-25T23:29:52.139692+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Received healthy response to inference request in 5.605644941329956s
Received healthy response to inference request in 5.741964101791382s
Received healthy response to inference request in 5.833584547042847s
Received healthy response to inference request in 5.710143327713013s
Received healthy response to inference request in 5.505640745162964s
Received healthy response to inference request in 5.509475469589233s
Received healthy response to inference request in 5.4954915046691895s
Received healthy response to inference request in 5.448707818984985s
Received healthy response to inference request in 5.545978546142578s
Received healthy response to inference request in 5.497976779937744s
2026-03-25T23:30:52.319375+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v4
Received healthy response to inference request in 5.578545331954956s
Received healthy response to inference request in 5.584344863891602s
Received healthy response to inference request in 5.635495185852051s
Received healthy response to inference request in 5.666703939437866s
30 requests
0 failed requests
5th percentile: 5.426165616512298
10th percentile: 5.474583840370178
20th percentile: 5.497118091583252
30th percentile: 5.508325052261353
40th percentile: 5.550426578521728
50th percentile: 5.581445097923279
60th percentile: 5.6254009246826175
70th percentile: 5.646425843238831
80th percentile: 5.7107678890228275
90th percentile: 5.751126146316529
95th percentile: 5.87622743844986
99th percentile: 9.434479787349705
mean time: 5.767822178204854
Pipeline stage StressChecker completed in 506.24s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.30s
Shutdown handler de-registered
chaiml-merged-qwen-35-d_39140_v4 status is now deployed due to DeploymentManager action
chaiml-merged-qwen-35-d_39140_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-merged-qwen-35-d_39140_v4 status is now torndown due to DeploymentManager action