developer_uid: rirv938
submission_id: chaiml-qwen35-bobo-19k-_54377_v6
model_name: chaiml-qwen35-bobo-19k-_54377_v6
model_group: ChaiML/qwen35_bobo_19k-s
status: torndown
timestamp: 2026-03-27T10:21:57+00:00
num_battles: 11204
num_wins: 6059
celo_rating: 8483.13
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen35_bobo_19k-step4455-merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen35-bobo-19k-_54377_v6
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen35_bobo_19k-step4455-merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-24
win_ratio: 0.5407890039271689
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_start|>', 'You:', '###', '<|im_end|>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen35-bobo-19k-54377-v6-uploader
Waiting for job on chaiml-qwen35-bobo-19k-54377-v6-uploader to finish
chaiml-qwen35-bobo-19k-54377-v6-uploader: Using quantization_mode: none
chaiml-qwen35-bobo-19k-54377-v6-uploader: Downloading snapshot of ChaiML/qwen35_bobo_19k-step4455-merged...
chaiml-qwen35-bobo-19k-54377-v6-uploader: Downloaded in 18.313s
chaiml-qwen35-bobo-19k-54377-v6-uploader: Processed model ChaiML/qwen35_bobo_19k-step4455-merged in 39.133s
chaiml-qwen35-bobo-19k-54377-v6-uploader: creating bucket guanaco-vllm-models
chaiml-qwen35-bobo-19k-54377-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v6-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen35-bobo-19k-54377-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen35-bobo-19k-54377-v6-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen35-bobo-19k-54377-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v6-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v6-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v6-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v6-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v6-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen35-bobo-19k-54377-v6-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v6-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen35-bobo-19k-54377-v6-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen35-bobo-19k-54377-v6-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen35-bobo-19k-54377-v6-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/generation_config.json
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/README.md
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model.safetensors.index.json
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/.gitattributes
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/chat_template.jinja
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/config.json
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/preprocessor_config.json
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/tokenizer_config.json
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/args.json
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/processor_config.json
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/tokenizer.json
2026-03-24T06:58:08.769867+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00001-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00001-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00012-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00012-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00002-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00002-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00004-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00004-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00009-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00009-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00003-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00003-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00011-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00011-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00006-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00006-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00007-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00007-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00005-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00005-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00010-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00010-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v6-uploader: cp /dev/shm/model_output/model-00008-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v6/default/model-00008-of-00012.safetensors
Job chaiml-qwen35-bobo-19k-54377-v6-uploader completed after 72.85s with status: succeeded
Stopping job with name chaiml-qwen35-bobo-19k-54377-v6-uploader
Pipeline stage VLLMUploader completed in 73.30s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.71s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen35-bobo-19k-54377-v6
Waiting for inference service chaiml-qwen35-bobo-19k-54377-v6 to be ready
2026-03-24T06:59:08.866739+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
2026-03-24T07:00:08.956356+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
2026-03-24T07:01:09.050367+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
Inference service chaiml-qwen35-bobo-19k-54377-v6 ready after 170.4756419658661s
Pipeline stage VLLMDeployer completed in 171.01s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T07:02:09.141525+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 17.15859365463257s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T07:03:09.238295+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.5093376636505127s
Received healthy response to inference request in 2.6177592277526855s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.462257146835327s
Received healthy response to inference request in 2.7556862831115723s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T07:04:09.360949+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 14.399434089660645s
Received healthy response to inference request in 2.532567262649536s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.5557563304901123s
Received healthy response to inference request in 2.526193380355835s
Received healthy response to inference request in 2.633227825164795s
2026-03-24T07:05:09.457974+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
Received healthy response to inference request in 2.5233471393585205s
Received healthy response to inference request in 2.678421974182129s
Received healthy response to inference request in 2.538133144378662s
Received healthy response to inference request in 2.4377048015594482s
Received healthy response to inference request in 9.372748374938965s
Received healthy response to inference request in 2.4961156845092773s
Received healthy response to inference request in 2.5837137699127197s
Received healthy response to inference request in 2.40431809425354s
Received healthy response to inference request in 2.5534772872924805s
Received healthy response to inference request in 2.5134615898132324s
Received healthy response to inference request in 2.6038284301757812s
30 requests
9 failed requests
5th percentile: 2.4487533569335938
10th percentile: 2.492729830741882
20th percentile: 2.521370029449463
30th percentile: 2.5364633798599243
40th percentile: 2.5725307941436766
50th percentile: 2.6254935264587402
60th percentile: 5.40251111984252
70th percentile: 18.0466417312622
80th percentile: 20.12175135612488
90th percentile: 20.136234760284424
95th percentile: 20.141698372364043
99th percentile: 20.150247192382814
mean time: 8.934710836410522
%s, retrying in %s seconds...
Received healthy response to inference request in 2.5677263736724854s
Received healthy response to inference request in 2.4774978160858154s
Received healthy response to inference request in 2.4977428913116455s
Received healthy response to inference request in 2.454712152481079s
Received healthy response to inference request in 2.4544100761413574s
Received healthy response to inference request in 2.3217742443084717s
Received healthy response to inference request in 2.3065357208251953s
Received healthy response to inference request in 2.661297559738159s
Received healthy response to inference request in 2.5606625080108643s
2026-03-24T07:06:09.558628+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v6
Received healthy response to inference request in 2.525867223739624s
Received healthy response to inference request in 2.483654737472534s
Received healthy response to inference request in 2.4677209854125977s
Received healthy response to inference request in 2.4959089756011963s
Received healthy response to inference request in 2.4800124168395996s
Received healthy response to inference request in 2.3883681297302246s
Received healthy response to inference request in 2.517072916030884s
Received healthy response to inference request in 2.6090641021728516s
Received healthy response to inference request in 2.4562151432037354s
Received healthy response to inference request in 2.534696102142334s
Received healthy response to inference request in 2.6236252784729004s
Received healthy response to inference request in 2.926628351211548s
Received healthy response to inference request in 2.5241243839263916s
Received healthy response to inference request in 2.5651652812957764s
Received healthy response to inference request in 2.551004648208618s
Received healthy response to inference request in 2.4319510459899902s
Received healthy response to inference request in 2.4862728118896484s
Received healthy response to inference request in 2.527498245239258s
Received healthy response to inference request in 2.570521831512451s
Received healthy response to inference request in 2.5430405139923096s
Received healthy response to inference request in 2.5687191486358643s
30 requests
0 failed requests
5th percentile: 2.3517414927482605
10th percentile: 2.4275927543640137
20th percentile: 2.455914545059204
30th percentile: 2.4792580366134644
40th percentile: 2.492054510116577
50th percentile: 2.5205986499786377
60th percentile: 2.5303773880004883
70th percentile: 2.553902006149292
80th percentile: 2.5679249286651613
90th percentile: 2.6105202198028565
95th percentile: 2.6443450331687925
99th percentile: 2.8496824216842653
mean time: 2.519316387176514
Pipeline stage StressChecker completed in 348.65s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.76s
Shutdown handler de-registered
chaiml-qwen35-bobo-19k-_54377_v6 status is now deployed due to DeploymentManager action
chaiml-qwen35-bobo-19k-_54377_v6 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen35-bobo-19k-_54377_v6 status is now torndown due to DeploymentManager action