developer_uid: rirv938
submission_id: chaiml-merged-qwen-35-d_39140_v5
model_name: chaiml-merged-qwen-35-d_39140_v5
model_group: ChaiML/merged_qwen_35_dp
status: torndown
timestamp: 2026-03-30T07:31:42+00:00
num_battles: 10506
num_wins: 5973
celo_rating: 1400.06
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/merged_qwen_35_dpo_lower_lr_v
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-merged-qwen-35-d_39140_v5
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/merged_qwen_35_dpo_lower_lr_v
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5685322672758424
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['###', '<|im_start|>', 'You:', '</s>', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-merged-qwen-35-d-39140-v5-uploader
Waiting for job on chaiml-merged-qwen-35-d-39140-v5-uploader to finish
chaiml-merged-qwen-35-d-39140-v5-uploader: Using quantization_mode: none
chaiml-merged-qwen-35-d-39140-v5-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v...
chaiml-merged-qwen-35-d-39140-v5-uploader: Downloaded in 33.196s
2026-03-27T04:39:01.473790+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
chaiml-merged-qwen-35-d-39140-v5-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 53.819s
chaiml-merged-qwen-35-d-39140-v5-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-d-39140-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v5-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-d-39140-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-d-39140-v5-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-d-39140-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v5-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-d-39140-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v5-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-d-39140-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v5-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-d-39140-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v5-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-d-39140-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-d-39140-v5-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-d-39140-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-d-39140-v5-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-d-39140-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-merged-qwen-35-d-39140-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/.gitattributes
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/config.json
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/README.md
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/generation_config.json
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/model.safetensors.index.json
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/tokenizer_config.json
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/tokenizer.json
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/model-00002-of-00002.safetensors
2026-03-27T04:40:01.568815+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
chaiml-merged-qwen-35-d-39140-v5-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v5/default/model-00001-of-00002.safetensors
Job chaiml-merged-qwen-35-d-39140-v5-uploader completed after 152.83s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-d-39140-v5-uploader
Pipeline stage VLLMUploader completed in 159.88s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.95s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-merged-qwen-35-d-39140-v5
Waiting for inference service chaiml-merged-qwen-35-d-39140-v5 to be ready
2026-03-27T04:41:01.779975+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
2026-03-27T04:42:01.876602+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
2026-03-27T04:43:01.974756+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
Inference service chaiml-merged-qwen-35-d-39140-v5 ready after 180.4000997543335s
Pipeline stage VLLMDeployer completed in 180.93s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-27T04:44:02.096413+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T04:45:02.218918+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
Received healthy response to inference request in 16.919103622436523s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.490077018737793s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T04:46:02.318928+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.473741292953491s
Received healthy response to inference request in 2.522559642791748s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.77209997177124s
Received healthy response to inference request in 2.6369433403015137s
Received healthy response to inference request in 2.558281183242798s
2026-03-27T04:47:02.429235+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.717519521713257s
Received healthy response to inference request in 2.71355938911438s
Received healthy response to inference request in 2.6464130878448486s
Received healthy response to inference request in 9.548889636993408s
Received healthy response to inference request in 2.5933616161346436s
Received healthy response to inference request in 2.926971912384033s
2026-03-27T04:48:02.523396+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.6371235847473145s
Received healthy response to inference request in 2.5909059047698975s
Received healthy response to inference request in 2.929609537124634s
Received healthy response to inference request in 9.40434217453003s
Received healthy response to inference request in 2.6907894611358643s
Received healthy response to inference request in 2.5941696166992188s
Received healthy response to inference request in 2.6470282077789307s
Received healthy response to inference request in 2.680666208267212s
30 requests
9 failed requests
5th percentile: 2.5386343359947205
10th percentile: 2.5876434326171873
20th percentile: 2.628388595581055
30th percentile: 2.646843671798706
40th percentile: 2.7044514179229737
50th percentile: 2.9282907247543335
60th percentile: 9.503800630569458
70th percentile: 17.875810527801505
80th percentile: 20.120595264434815
90th percentile: 20.13413460254669
95th percentile: 20.148500418663026
99th percentile: 24.135951559543614
mean time: 9.48254845937093
%s, retrying in %s seconds...
Received healthy response to inference request in 2.6004295349121094s
Received healthy response to inference request in 2.5255513191223145s
Received healthy response to inference request in 2.3847639560699463s
Received healthy response to inference request in 2.5524580478668213s
Received healthy response to inference request in 2.9472668170928955s
Received healthy response to inference request in 2.6050631999969482s
Received healthy response to inference request in 2.478689432144165s
Received healthy response to inference request in 2.523895263671875s
Received healthy response to inference request in 2.655646324157715s
Received healthy response to inference request in 2.5150489807128906s
Received healthy response to inference request in 2.477943181991577s
2026-03-27T04:49:02.619648+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v5
Received healthy response to inference request in 2.512516975402832s
Received healthy response to inference request in 2.521618604660034s
Received healthy response to inference request in 2.6405160427093506s
Received healthy response to inference request in 2.5489604473114014s
Received healthy response to inference request in 2.5632081031799316s
Received healthy response to inference request in 2.560702323913574s
Received healthy response to inference request in 2.6244139671325684s
Received healthy response to inference request in 2.59004807472229s
Received healthy response to inference request in 2.6379737854003906s
Received healthy response to inference request in 2.6255204677581787s
Received healthy response to inference request in 2.5826807022094727s
Received healthy response to inference request in 2.679628372192383s
Received healthy response to inference request in 2.5750880241394043s
Received healthy response to inference request in 2.879056692123413s
Received healthy response to inference request in 2.6062633991241455s
Received healthy response to inference request in 2.574031114578247s
Received healthy response to inference request in 2.712116241455078s
Received healthy response to inference request in 2.696993589401245s
Received healthy response to inference request in 2.5706229209899902s
30 requests
0 failed requests
5th percentile: 2.4782789945602417
10th percentile: 2.509134221076965
20th percentile: 2.5234399318695067
30th percentile: 2.5514087677001953
40th percentile: 2.567656993865967
50th percentile: 2.5788843631744385
60th percentile: 2.602283000946045
70th percentile: 2.6247459173202516
80th percentile: 2.6435420989990233
90th percentile: 2.6985058546066285
95th percentile: 2.8039334893226617
99th percentile: 2.9274858808517457
mean time: 2.5989571968714396
Pipeline stage StressChecker completed in 367.02s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-merged-qwen-35-d_39140_v5 status is now deployed due to DeploymentManager action
chaiml-merged-qwen-35-d_39140_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-merged-qwen-35-d_39140_v5 status is now torndown due to DeploymentManager action