developer_uid: rirv938
submission_id: chaiml-qwen35-bobo-19k-_54377_v7
model_name: chaiml-qwen35-bobo-19k-_54377_v7
model_group: ChaiML/qwen35_bobo_19k-s
status: torndown
timestamp: 2026-03-28T01:01:56+00:00
num_battles: 10216
num_wins: 5488
celo_rating: 1318.26
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen35_bobo_19k-step4455-merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen35-bobo-19k-_54377_v7
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen35_bobo_19k-step4455-merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-24
win_ratio: 0.5371965544244323
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '###', '</s>', '<|im_start|>', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen35-bobo-19k-54377-v7-uploader
Waiting for job on chaiml-qwen35-bobo-19k-54377-v7-uploader to finish
chaiml-qwen35-bobo-19k-54377-v7-uploader: Using quantization_mode: none
chaiml-qwen35-bobo-19k-54377-v7-uploader: Downloading snapshot of ChaiML/qwen35_bobo_19k-step4455-merged...
chaiml-qwen35-bobo-19k-54377-v7-uploader: Downloaded in 20.223s
chaiml-qwen35-bobo-19k-54377-v7-uploader: Processed model ChaiML/qwen35_bobo_19k-step4455-merged in 40.464s
2026-03-24T19:21:23.463448+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
chaiml-qwen35-bobo-19k-54377-v7-uploader: creating bucket guanaco-vllm-models
chaiml-qwen35-bobo-19k-54377-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen35-bobo-19k-54377-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen35-bobo-19k-54377-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen35-bobo-19k-54377-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen35-bobo-19k-54377-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen35-bobo-19k-54377-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen35-bobo-19k-54377-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen35-bobo-19k-54377-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/.gitattributes
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/args.json
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/chat_template.jinja
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/config.json
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/preprocessor_config.json
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/tokenizer_config.json
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/generation_config.json
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/README.md
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/processor_config.json
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/model.safetensors.index.json
chaiml-qwen35-bobo-19k-54377-v7-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v7/default/tokenizer.json
Job chaiml-qwen35-bobo-19k-54377-v7-uploader completed after 74.29s with status: succeeded
Stopping job with name chaiml-qwen35-bobo-19k-54377-v7-uploader
Pipeline stage VLLMUploader completed in 80.05s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.34s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen35-bobo-19k-54377-v7
Waiting for inference service chaiml-qwen35-bobo-19k-54377-v7 to be ready
2026-03-24T19:22:26.739475+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
2026-03-24T19:23:26.839802+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
2026-03-24T19:24:26.923568+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
Inference service chaiml-qwen35-bobo-19k-54377-v7 ready after 180.77705097198486s
Pipeline stage VLLMDeployer completed in 181.33s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T19:25:27.017735+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 17.144242763519287s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T19:26:27.108342+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T19:27:27.195834+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
Received healthy response to inference request in 2.661461591720581s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.615475177764893s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T19:28:27.282975+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
Received healthy response to inference request in 9.326599836349487s
Received healthy response to inference request in 2.4234392642974854s
Received healthy response to inference request in 2.635352611541748s
Received healthy response to inference request in 2.913142442703247s
Received healthy response to inference request in 2.4504151344299316s
Received healthy response to inference request in 2.644737482070923s
Received healthy response to inference request in 2.879397392272949s
Received healthy response to inference request in 2.6334567070007324s
Received healthy response to inference request in 9.493740320205688s
Received healthy response to inference request in 2.6002135276794434s
Received healthy response to inference request in 2.4591870307922363s
Received healthy response to inference request in 2.5854923725128174s
Received healthy response to inference request in 2.471679925918579s
Received healthy response to inference request in 2.5408122539520264s
Received healthy response to inference request in 2.7087085247039795s
Received healthy response to inference request in 2.6810383796691895s
Received healthy response to inference request in 2.5387423038482666s
Received healthy response to inference request in 2.499276638031006s
30 requests
9 failed requests
5th percentile: 2.454362487792969
10th percentile: 2.4704306364059447
20th percentile: 2.5403982639312743
30th percentile: 2.623483753204346
40th percentile: 2.6547719478607177
50th percentile: 2.7940529584884644
60th percentile: 9.393456029891967
70th percentile: 18.035750603675833
80th percentile: 20.12258005142212
90th percentile: 20.126935911178588
95th percentile: 20.13386380672455
99th percentile: 20.18562427997589
mean time: 9.037081678708395
%s, retrying in %s seconds...
Received healthy response to inference request in 2.5666608810424805s
Received healthy response to inference request in 2.537796974182129s
2026-03-24T19:29:27.381332+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
Received healthy response to inference request in 2.5522820949554443s
Received healthy response to inference request in 2.4485323429107666s
Received healthy response to inference request in 2.85931658744812s
Received healthy response to inference request in 2.499422788619995s
Received healthy response to inference request in 2.433117389678955s
Received healthy response to inference request in 2.3757407665252686s
Received healthy response to inference request in 2.859403371810913s
Received healthy response to inference request in 2.542696475982666s
Received healthy response to inference request in 2.6715588569641113s
Received healthy response to inference request in 2.5134687423706055s
Received healthy response to inference request in 2.3897502422332764s
Received healthy response to inference request in 2.526874542236328s
Received healthy response to inference request in 2.631938934326172s
Received healthy response to inference request in 2.53438663482666s
Received healthy response to inference request in 2.6344780921936035s
Received healthy response to inference request in 2.4564478397369385s
Received healthy response to inference request in 2.7538092136383057s
Received healthy response to inference request in 2.792506217956543s
Received healthy response to inference request in 2.535837411880493s
2026-03-24T19:30:27.479263+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v7
Received healthy response to inference request in 2.4862804412841797s
Received healthy response to inference request in 2.635646104812622s
Received healthy response to inference request in 2.5696849822998047s
Received healthy response to inference request in 2.8109657764434814s
Received healthy response to inference request in 2.5962438583374023s
Received healthy response to inference request in 2.549013614654541s
Received healthy response to inference request in 2.518794536590576s
Received healthy response to inference request in 2.7692911624908447s
Received healthy response to inference request in 2.6746058464050293s
30 requests
0 failed requests
5th percentile: 2.409265458583832
10th percentile: 2.4469908475875854
20th percentile: 2.496794319152832
30th percentile: 2.5244505405426025
40th percentile: 2.5370131492614747
50th percentile: 2.5506478548049927
60th percentile: 2.580308532714844
70th percentile: 2.634828495979309
80th percentile: 2.690446519851685
90th percentile: 2.794352173805237
95th percentile: 2.8375587224960324
99th percentile: 2.859378204345703
mean time: 2.5908850908279417
Pipeline stage StressChecker completed in 368.12s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.71s
Shutdown handler de-registered
chaiml-qwen35-bobo-19k-_54377_v7 status is now deployed due to DeploymentManager action
chaiml-qwen35-bobo-19k-_54377_v7 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen35-bobo-19k-_54377_v7 status is now torndown due to DeploymentManager action