developer_uid: chai_backend_admin
submission_id: chaiml-merged-qwen-35-_39140_v13
model_name: chaiml-merged-qwen-35-_39140_v13
model_group: ChaiML/merged_qwen_35_dp
status: torndown
timestamp: 2026-03-31T00:21:48+00:00
num_battles: 11484
num_wins: 6420
celo_rating: 1336.97
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/merged_qwen_35_dpo_lower_lr_v
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-merged-qwen-35-_39140_v13
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/merged_qwen_35_dpo_lower_lr_v
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5590386624869383
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['###', '<|im_end|>', '</s>', '<|im_start|>', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-merged-qwen-35-39140-v13-uploader
Waiting for job on chaiml-merged-qwen-35-39140-v13-uploader to finish
chaiml-merged-qwen-35-39140-v13-uploader: Using quantization_mode: fp8
chaiml-merged-qwen-35-39140-v13-uploader: Checking if ChaiML/merged_qwen_35_dpo_lower_lr_v-FP8 already exists in ChaiML
chaiml-merged-qwen-35-39140-v13-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-merged-qwen-35-39140-v13-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v-FP8...
chaiml-merged-qwen-35-39140-v13-uploader: Downloaded in 30.981s
chaiml-merged-qwen-35-39140-v13-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 33.498s
chaiml-merged-qwen-35-39140-v13-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-39140-v13-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v13-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-39140-v13-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-39140-v13-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-39140-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v13-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v13-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v13-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v13-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v13-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-39140-v13-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-39140-v13-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-39140-v13-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-39140-v13-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-merged-qwen-35-39140-v13-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default
chaiml-merged-qwen-35-39140-v13-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default/chat_template.jinja
chaiml-merged-qwen-35-39140-v13-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default/recipe.yaml
chaiml-merged-qwen-35-39140-v13-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default/generation_config.json
2026-03-27T22:55:31.503761+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
chaiml-merged-qwen-35-39140-v13-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default/.gitattributes
chaiml-merged-qwen-35-39140-v13-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default/tokenizer_config.json
chaiml-merged-qwen-35-39140-v13-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default/config.json
chaiml-merged-qwen-35-39140-v13-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default/tokenizer.json
chaiml-merged-qwen-35-39140-v13-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v13/default/model.safetensors
Job chaiml-merged-qwen-35-39140-v13-uploader completed after 109.57s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-39140-v13-uploader
Pipeline stage VLLMUploader completed in 110.87s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.32s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-merged-qwen-35-39140-v13
Waiting for inference service chaiml-merged-qwen-35-39140-v13 to be ready
2026-03-27T22:56:31.608516+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
2026-03-27T22:57:31.701945+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
2026-03-27T22:58:31.832593+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
Inference service chaiml-merged-qwen-35-39140-v13 ready after 160.41104769706726s
Pipeline stage VLLMDeployer completed in 160.95s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T22:59:32.284395+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 18.155683040618896s
2026-03-27T23:00:32.449569+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 5.094224214553833s
Received healthy response to inference request in 10.336251020431519s
Received healthy response to inference request in 10.422263383865356s
Received healthy response to inference request in 5.050529956817627s
2026-03-27T23:01:32.627811+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 5.036867380142212s
Received healthy response to inference request in 10.446770668029785s
Received healthy response to inference request in 5.144803762435913s
Received healthy response to inference request in 6.01332688331604s
Received healthy response to inference request in 5.244468450546265s
Received healthy response to inference request in 5.260451555252075s
Received healthy response to inference request in 5.479774236679077s
Received healthy response to inference request in 5.255897283554077s
Received healthy response to inference request in 5.479609489440918s
2026-03-27T23:02:32.771905+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
Received healthy response to inference request in 10.422235012054443s
Received healthy response to inference request in 5.095623731613159s
Received healthy response to inference request in 5.199480772018433s
Received healthy response to inference request in 5.464714527130127s
Received healthy response to inference request in 5.029480457305908s
Received healthy response to inference request in 5.405047416687012s
Received healthy response to inference request in 5.030759811401367s
Received healthy response to inference request in 5.169384956359863s
Received healthy response to inference request in 5.546241760253906s
Received healthy response to inference request in 5.237952947616577s
2026-03-27T23:03:32.871775+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
Received healthy response to inference request in 5.074105501174927s
30 requests
5 failed requests
5th percentile: 5.033508217334747
10th percentile: 5.0491636991500854
20th percentile: 5.095343828201294
30th percentile: 5.190452027320862
40th percentile: 5.2513257503509525
50th percentile: 5.434880971908569
60th percentile: 5.506361246109009
70th percentile: 10.362046217918396
80th percentile: 11.98855314254763
90th percentile: 20.119372248649597
95th percentile: 20.133639419078825
99th percentile: 20.1574010181427
mean time: 8.858699727058411
%s, retrying in %s seconds...
Failed to get request counts for guanaco-submitter. Falling back to default
Received healthy response to inference request in 5.099523305892944s
Received healthy response to inference request in 5.4447267055511475s
Received healthy response to inference request in 4.989161014556885s
Received healthy response to inference request in 5.1790244579315186s
Received healthy response to inference request in 5.086450576782227s
Received healthy response to inference request in 5.028339862823486s
Received healthy response to inference request in 5.140702486038208s
Received healthy response to inference request in 4.982517242431641s
Received healthy response to inference request in 5.5694427490234375s
Received healthy response to inference request in 5.010790586471558s
2026-03-27T23:04:32.982128+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
Received healthy response to inference request in 5.017691373825073s
Received healthy response to inference request in 5.17986798286438s
Received healthy response to inference request in 5.013011455535889s
Received healthy response to inference request in 5.683335065841675s
Received healthy response to inference request in 5.123686075210571s
Received healthy response to inference request in 5.402713298797607s
Received healthy response to inference request in 5.048395156860352s
Received healthy response to inference request in 5.157573699951172s
Received healthy response to inference request in 4.9258270263671875s
Received healthy response to inference request in 5.187433242797852s
Received healthy response to inference request in 5.1105592250823975s
Received healthy response to inference request in 5.309884786605835s
2026-03-27T23:05:33.485181+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v13
Received healthy response to inference request in 4.88914942741394s
Received healthy response to inference request in 5.055727243423462s
Received healthy response to inference request in 5.157749652862549s
Received healthy response to inference request in 5.050195217132568s
Received healthy response to inference request in 5.062367677688599s
Received healthy response to inference request in 4.986773252487183s
Received healthy response to inference request in 5.543945550918579s
Received healthy response to inference request in 5.0975096225738525s
30 requests
0 failed requests
5th percentile: 4.951337623596191
10th percentile: 4.986347651481628
20th percentile: 5.0125672817230225
30th percentile: 5.042378568649292
40th percentile: 5.059711503982544
50th percentile: 5.098516464233398
60th percentile: 5.130492639541626
70th percentile: 5.164132094383239
80th percentile: 5.211923551559448
90th percentile: 5.454648590087891
95th percentile: 5.557969009876251
99th percentile: 5.650306293964386
mean time: 5.151135834058126
Pipeline stage StressChecker completed in 426.51s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.70s
Shutdown handler de-registered
chaiml-merged-qwen-35-_39140_v13 status is now deployed due to DeploymentManager action
chaiml-merged-qwen-35-_39140_v13 status is now inactive due to auto deactivation removed underperforming models
chaiml-merged-qwen-35-_39140_v13 status is now torndown due to DeploymentManager action