chaiml-merged-qwen-35-_39140

developer_uid: chai_backend_admin

submission_id: chaiml-merged-qwen-35-_39140_v14

model_name: chaiml-merged-qwen-35-_39140_v14

model_group: ChaiML/merged_qwen_35_dp

status: torndown

timestamp: 2026-03-31T00:31:38+00:00

num_battles: 12021

num_wins: 6798

celo_rating: 1338.37

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/merged_qwen_35_dpo_lower_lr_v

model_architecture: Qwen3_5ForConditionalGeneration

model_num_parameters: 23564784640.0

best_of: 8

max_input_tokens: 2048

max_output_tokens: 80

reward_model: default

display_name: chaiml-merged-qwen-35-_39140_v14

ineligible_reason: max_output_tokens!=64

is_internal_developer: True

language_model: ChaiML/merged_qwen_35_dpo_lower_lr_v

model_size: 24B

ranking_group: single

us_pacific_date: 2026-03-27

win_ratio: 0.5655103568754679

generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '###', '<|im_end|>', '</s>', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}

formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-merged-qwen-35-39140-v14-uploader
Waiting for job on chaiml-merged-qwen-35-39140-v14-uploader to finish
chaiml-merged-qwen-35-39140-v14-uploader: Using quantization_mode: fp8
chaiml-merged-qwen-35-39140-v14-uploader: Checking if ChaiML/merged_qwen_35_dpo_lower_lr_v-FP8 already exists in ChaiML
chaiml-merged-qwen-35-39140-v14-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-merged-qwen-35-39140-v14-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v-FP8...
2026-03-27T22:55:31.915933+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
chaiml-merged-qwen-35-39140-v14-uploader: Downloaded in 26.786s
chaiml-merged-qwen-35-39140-v14-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 29.260s
chaiml-merged-qwen-35-39140-v14-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-39140-v14-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v14-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-39140-v14-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-39140-v14-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-39140-v14-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v14-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v14-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v14-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v14-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v14-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v14-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v14-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v14-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-39140-v14-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-39140-v14-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-39140-v14-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-39140-v14-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-merged-qwen-35-39140-v14-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default
chaiml-merged-qwen-35-39140-v14-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default/.gitattributes
chaiml-merged-qwen-35-39140-v14-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default/chat_template.jinja
chaiml-merged-qwen-35-39140-v14-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default/tokenizer_config.json
chaiml-merged-qwen-35-39140-v14-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default/recipe.yaml
chaiml-merged-qwen-35-39140-v14-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default/generation_config.json
chaiml-merged-qwen-35-39140-v14-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default/config.json
chaiml-merged-qwen-35-39140-v14-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default/tokenizer.json
2026-03-27T22:56:32.411679+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
chaiml-merged-qwen-35-39140-v14-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v14/default/model.safetensors
Job chaiml-merged-qwen-35-39140-v14-uploader completed after 138.12s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-39140-v14-uploader
Pipeline stage VLLMUploader completed in 138.65s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.12s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-merged-qwen-35-39140-v14
Waiting for inference service chaiml-merged-qwen-35-39140-v14 to be ready
2026-03-27T22:57:32.511497+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
2026-03-27T22:58:32.625245+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
2026-03-27T22:59:32.741834+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
Inference service chaiml-merged-qwen-35-39140-v14 ready after 170.46703219413757s
Pipeline stage VLLMDeployer completed in 170.95s
run pipeline stage %s
Running pipeline stage StressChecker
Failed to get request counts for guanaco-submitter. Falling back to default
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T23:00:32.892297+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 17.260616779327393s
2026-03-27T23:01:32.989612+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
Received healthy response to inference request in 10.459395408630371s
Received healthy response to inference request in 5.112008571624756s
Received healthy response to inference request in 5.935158014297485s
Received healthy response to inference request in 10.673008680343628s
Received healthy response to inference request in 5.2749269008636475s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T23:02:33.083433+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 18.531604766845703s
Received healthy response to inference request in 5.228705167770386s
Received healthy response to inference request in 5.263564348220825s
Received healthy response to inference request in 5.681760787963867s
Received healthy response to inference request in 10.74279260635376s
Received healthy response to inference request in 5.220145225524902s
2026-03-27T23:03:33.183139+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
Received healthy response to inference request in 5.3086559772491455s
Received healthy response to inference request in 5.493555068969727s
Received healthy response to inference request in 5.359590530395508s
Received healthy response to inference request in 5.356555700302124s
Received healthy response to inference request in 5.29700231552124s
Received healthy response to inference request in 5.275213003158569s
Received healthy response to inference request in 5.329375743865967s
Received healthy response to inference request in 5.323257923126221s
Received healthy response to inference request in 5.344231605529785s
Received healthy response to inference request in 5.919112920761108s
2026-03-27T23:04:33.426549+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
Received healthy response to inference request in 5.473340749740601s
Received healthy response to inference request in 5.394292116165161s
30 requests
6 failed requests
5th percentile: 5.22399719953537
10th percentile: 5.260078430175781
20th percentile: 5.292644453048706
30th percentile: 5.327540397644043
40th percentile: 5.358376598358154
50th percentile: 5.483447909355164
60th percentile: 5.925530958175659
70th percentile: 10.693943858146667
80th percentile: 18.84868841171265
90th percentile: 20.139470553398134
95th percentile: 20.20625309944153
99th percentile: 20.85126148223877
mean time: 9.738009810447693
%s, retrying in %s seconds...
Received healthy response to inference request in 5.091381072998047s
Received healthy response to inference request in 5.0942747592926025s
Received healthy response to inference request in 5.197667121887207s
Received healthy response to inference request in 5.086905002593994s
Received healthy response to inference request in 5.129175662994385s
Received healthy response to inference request in 5.068754434585571s
Received healthy response to inference request in 5.0866522789001465s
Received healthy response to inference request in 5.119096994400024s
Received healthy response to inference request in 5.122450828552246s
Received healthy response to inference request in 5.131364583969116s
2026-03-27T23:05:33.537791+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
Received healthy response to inference request in 5.208230972290039s
Received healthy response to inference request in 5.325353384017944s
Received healthy response to inference request in 5.214561939239502s
Received healthy response to inference request in 5.47145414352417s
Received healthy response to inference request in 5.240122079849243s
Received healthy response to inference request in 5.45354700088501s
Received healthy response to inference request in 5.750173568725586s
Received healthy response to inference request in 5.212586879730225s
Received healthy response to inference request in 5.653918743133545s
Received healthy response to inference request in 5.260289907455444s
Received healthy response to inference request in 5.497233867645264s
2026-03-27T23:06:33.637421+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v14
Received healthy response to inference request in 5.480900526046753s
Received healthy response to inference request in 5.2658531665802s
Received healthy response to inference request in 5.54271936416626s
Received healthy response to inference request in 5.635693550109863s
Received healthy response to inference request in 5.421460390090942s
Received healthy response to inference request in 5.512811660766602s
Received healthy response to inference request in 5.281905889511108s
Received healthy response to inference request in 5.548713684082031s
Received healthy response to inference request in 5.392891883850098s
30 requests
0 failed requests
5th percentile: 5.086766004562378
10th percentile: 5.090933465957642
20th percentile: 5.121780061721802
30th percentile: 5.17777636051178
40th percentile: 5.213771915435791
50th percentile: 5.263071537017822
60th percentile: 5.352368783950806
70th percentile: 5.4589191436767575
80th percentile: 5.500349426269532
90th percentile: 5.5574116706848145
95th percentile: 5.6457174062728885
99th percentile: 5.722259669303894
mean time: 5.316604844729105
Pipeline stage StressChecker completed in 457.74s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.60s
Shutdown handler de-registered
chaiml-merged-qwen-35-_39140_v14 status is now deployed due to DeploymentManager action
chaiml-merged-qwen-35-_39140_v14 status is now inactive due to auto deactivation removed underperforming models
chaiml-merged-qwen-35-_39140_v14 status is now torndown due to DeploymentManager action