chaiml-qwen35-bobo-19k-_54377

developer_uid: rirv938
submission_id: chaiml-qwen35-bobo-19k-_54377_v5
model_name: chaiml-qwen35-bobo-19k-_54377_v5
model_group: ChaiML/qwen35_bobo_19k-s
status: torndown
timestamp: 2026-03-27T05:21:38+00:00
num_battles: 10418
num_wins: 5808
celo_rating: 1340.13
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen35_bobo_19k-step4455-merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen35-bobo-19k-_54377_v5
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen35_bobo_19k-step4455-merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-23
win_ratio: 0.557496640430025
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '<|im_start|>', 'You:', '</s>', '###'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen35-bobo-19k-54377-v5-uploader
Waiting for job on chaiml-qwen35-bobo-19k-54377-v5-uploader to finish
chaiml-qwen35-bobo-19k-54377-v5-uploader: Using quantization_mode: none
chaiml-qwen35-bobo-19k-54377-v5-uploader: Downloading snapshot of ChaiML/qwen35_bobo_19k-step4455-merged...
chaiml-qwen35-bobo-19k-54377-v5-uploader: Downloaded in 18.760s
chaiml-qwen35-bobo-19k-54377-v5-uploader: Processed model ChaiML/qwen35_bobo_19k-step4455-merged in 39.193s
chaiml-qwen35-bobo-19k-54377-v5-uploader: creating bucket guanaco-vllm-models
chaiml-qwen35-bobo-19k-54377-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v5-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen35-bobo-19k-54377-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen35-bobo-19k-54377-v5-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen35-bobo-19k-54377-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v5-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v5-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v5-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v5-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen35-bobo-19k-54377-v5-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen35-bobo-19k-54377-v5-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen35-bobo-19k-54377-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen35-bobo-19k-54377-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/processor_config.json
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/.gitattributes
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model.safetensors.index.json
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/chat_template.jinja
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/preprocessor_config.json
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/tokenizer_config.json
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/README.md
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/generation_config.json
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/config.json
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/args.json
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/tokenizer.json
2026-03-24T02:27:16.302336+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00001-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00001-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00012-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00012-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00002-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00002-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00007-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00007-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00004-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00004-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00006-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00006-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00010-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00010-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00008-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00008-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00011-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00011-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00005-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00005-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00009-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00009-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v5-uploader: cp /dev/shm/model_output/model-00003-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v5/default/model-00003-of-00012.safetensors
Job chaiml-qwen35-bobo-19k-54377-v5-uploader completed after 73.09s with status: succeeded
Stopping job with name chaiml-qwen35-bobo-19k-54377-v5-uploader
Pipeline stage VLLMUploader completed in 73.56s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.75s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen35-bobo-19k-54377-v5
Waiting for inference service chaiml-qwen35-bobo-19k-54377-v5 to be ready
2026-03-24T02:28:16.401918+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
2026-03-24T02:29:16.497851+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
2026-03-24T02:30:16.600473+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
Inference service chaiml-qwen35-bobo-19k-54377-v5 ready after 180.44785976409912s
Pipeline stage VLLMDeployer completed in 181.00s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T02:31:16.702485+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T02:32:16.803212+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 10.139300107955933s
Received healthy response to inference request in 9.353523969650269s
Received healthy response to inference request in 9.341275453567505s
2026-03-24T02:33:16.903494+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.639409303665161s
Received healthy response to inference request in 2.587085485458374s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.5377204418182373s
2026-03-24T02:34:17.006735+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
Received healthy response to inference request in 12.156412839889526s
Received healthy response to inference request in 2.415278673171997s
Received healthy response to inference request in 2.595803737640381s
Received healthy response to inference request in 2.429070234298706s
Received healthy response to inference request in 2.770975112915039s
Received healthy response to inference request in 2.6138486862182617s
Received healthy response to inference request in 2.694272041320801s
Received healthy response to inference request in 2.6184418201446533s
Received healthy response to inference request in 2.542715311050415s
Received healthy response to inference request in 2.6713051795959473s
Received healthy response to inference request in 2.6146187782287598s
Received healthy response to inference request in 2.7404375076293945s
Received healthy response to inference request in 2.626673698425293s
Received healthy response to inference request in 2.5095832347869873s
Received healthy response to inference request in 2.577235698699951s
30 requests
9 failed requests
5th percentile: 2.4653010845184324
10th percentile: 2.5349067211151124
20th percentile: 2.5851155281066895
30th percentile: 2.6143877506256104
40th percentile: 2.6343150615692137
50th percentile: 2.7173547744750977
60th percentile: 9.34617486000061
70th percentile: 14.549007582664467
80th percentile: 20.136488676071167
90th percentile: 20.14888391494751
95th percentile: 20.15623422861099
99th percentile: 20.169332394599916
mean time: 8.883240135510762
%s, retrying in %s seconds...
Received healthy response to inference request in 10.260039567947388s
Received healthy response to inference request in 2.3805086612701416s
Received healthy response to inference request in 2.4643795490264893s
2026-03-24T02:35:17.133455+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
Received healthy response to inference request in 3.6529619693756104s
Received healthy response to inference request in 2.730367422103882s
Received healthy response to inference request in 2.5082738399505615s
Received healthy response to inference request in 2.4597368240356445s
Received healthy response to inference request in 2.5247395038604736s
Received healthy response to inference request in 2.4024810791015625s
Received healthy response to inference request in 2.494767665863037s
Received healthy response to inference request in 2.8894786834716797s
Received healthy response to inference request in 2.6308541297912598s
Received healthy response to inference request in 2.437490940093994s
Received healthy response to inference request in 2.6408193111419678s
Received healthy response to inference request in 2.614234447479248s
Received healthy response to inference request in 2.75627064704895s
Received healthy response to inference request in 2.5523805618286133s
Received healthy response to inference request in 2.643845558166504s
Received healthy response to inference request in 2.4984982013702393s
Received healthy response to inference request in 2.4944515228271484s
Received healthy response to inference request in 2.5173230171203613s
Received healthy response to inference request in 2.5491249561309814s
Received healthy response to inference request in 2.5771398544311523s
Received healthy response to inference request in 2.4977757930755615s
Received healthy response to inference request in 2.6201894283294678s
2026-03-24T02:36:17.345078+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v5
Received healthy response to inference request in 2.534215211868286s
Received healthy response to inference request in 2.6281027793884277s
Received healthy response to inference request in 2.6629960536956787s
Received healthy response to inference request in 2.416454792022705s
Received healthy response to inference request in 2.5513839721679688s
30 requests
0 failed requests
5th percentile: 2.4087692499160767
10th percentile: 2.4353873252868654
20th percentile: 2.488437128067017
30th percentile: 2.498281478881836
40th percentile: 2.5217729091644285
50th percentile: 2.550254464149475
60th percentile: 2.5919776916503907
70th percentile: 2.6289281845092773
80th percentile: 2.647675657272339
90th percentile: 2.769591450691223
95th percentile: 3.3093944907188395
99th percentile: 8.343987064361578
mean time: 2.8530428647994994
Pipeline stage StressChecker completed in 359.39s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.24s
Shutdown handler de-registered
chaiml-qwen35-bobo-19k-_54377_v5 status is now deployed due to DeploymentManager action
chaiml-qwen35-bobo-19k-_54377_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen35-bobo-19k-_54377_v5 status is now torndown due to DeploymentManager action