developer_uid: rirv938
submission_id: chaiml-qwen35-bobo-19k-_54377_v8
model_name: chaiml-qwen35-bobo-19k-_54377_v8
model_group: ChaiML/qwen35_bobo_19k-s
status: torndown
timestamp: 2026-03-28T14:59:03+00:00
num_battles: 10136
num_wins: 5575
celo_rating: 1324.51
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen35_bobo_19k-step4455-merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen35-bobo-19k-_54377_v8
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen35_bobo_19k-step4455-merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-25
win_ratio: 0.5500197316495659
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '</s>', '<|im_end|>', '###', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen35-bobo-19k-54377-v8-uploader
Waiting for job on chaiml-qwen35-bobo-19k-54377-v8-uploader to finish
chaiml-qwen35-bobo-19k-54377-v8-uploader: Using quantization_mode: none
chaiml-qwen35-bobo-19k-54377-v8-uploader: Downloading snapshot of ChaiML/qwen35_bobo_19k-step4455-merged...
chaiml-qwen35-bobo-19k-54377-v8-uploader: Downloaded in 20.478s
chaiml-qwen35-bobo-19k-54377-v8-uploader: Processed model ChaiML/qwen35_bobo_19k-step4455-merged in 40.929s
chaiml-qwen35-bobo-19k-54377-v8-uploader: creating bucket guanaco-vllm-models
chaiml-qwen35-bobo-19k-54377-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v8-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen35-bobo-19k-54377-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen35-bobo-19k-54377-v8-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen35-bobo-19k-54377-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v8-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v8-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v8-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v8-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v8-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen35-bobo-19k-54377-v8-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v8-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen35-bobo-19k-54377-v8-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen35-bobo-19k-54377-v8-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen35-bobo-19k-54377-v8-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/.gitattributes
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/tokenizer_config.json
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/config.json
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/generation_config.json
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/chat_template.jinja
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/args.json
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model.safetensors.index.json
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/preprocessor_config.json
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/README.md
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/processor_config.json
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/tokenizer.json
2026-03-25T05:55:05.830965+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00001-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00001-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00012-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00012-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00004-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00004-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00002-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00002-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00010-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00010-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00003-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00003-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00007-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00007-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00011-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00011-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00009-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00009-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00008-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00008-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00006-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00006-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v8-uploader: cp /dev/shm/model_output/model-00005-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v8/default/model-00005-of-00012.safetensors
Job chaiml-qwen35-bobo-19k-54377-v8-uploader completed after 72.71s with status: succeeded
Stopping job with name chaiml-qwen35-bobo-19k-54377-v8-uploader
Pipeline stage VLLMUploader completed in 81.99s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.32s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen35-bobo-19k-54377-v8
Waiting for inference service chaiml-qwen35-bobo-19k-54377-v8 to be ready
2026-03-25T05:56:05.915270+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
2026-03-25T05:57:06.010343+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
2026-03-25T05:58:06.110890+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
Inference service chaiml-qwen35-bobo-19k-54377-v8 ready after 180.55252623558044s
Pipeline stage VLLMDeployer completed in 181.04s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T05:59:06.196914+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T06:00:06.344772+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 16.982181310653687s
Received healthy response to inference request in 9.121816396713257s
2026-03-25T06:01:06.740872+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.4286348819732666s
Received healthy response to inference request in 2.779738426208496s
Received healthy response to inference request in 9.566035032272339s
Received healthy response to inference request in 2.6579861640930176s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.6589267253875732s
Received healthy response to inference request in 2.5361409187316895s
2026-03-25T06:02:06.831512+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
Received healthy response to inference request in 14.505025863647461s
Received healthy response to inference request in 9.476255655288696s
Received healthy response to inference request in 2.5109975337982178s
Received healthy response to inference request in 2.5792438983917236s
Received healthy response to inference request in 2.5026779174804688s
Received healthy response to inference request in 2.929020643234253s
Received healthy response to inference request in 2.58510684967041s
Received healthy response to inference request in 2.4910473823547363s
Received healthy response to inference request in 2.6747093200683594s
Received healthy response to inference request in 2.6946005821228027s
Received healthy response to inference request in 2.58288836479187s
Received healthy response to inference request in 2.596275806427002s
Received healthy response to inference request in 2.6545679569244385s
Received healthy response to inference request in 2.431955337524414s
Received healthy response to inference request in 2.5384981632232666s
30 requests
7 failed requests
5th percentile: 2.458546757698059
10th percentile: 2.5015148639678957
20th percentile: 2.538026714324951
30th percentile: 2.584441304206848
40th percentile: 2.6566188812255858
50th percentile: 2.684654951095581
60th percentile: 5.406138944625846
70th percentile: 11.047732281684862
80th percentile: 20.104759693145752
90th percentile: 20.120162415504456
95th percentile: 20.12904485464096
99th percentile: 20.13316708803177
mean time: 8.243970370292663
%s, retrying in %s seconds...
Received healthy response to inference request in 2.330279588699341s
Received healthy response to inference request in 2.525510311126709s
Received healthy response to inference request in 2.688070058822632s
Received healthy response to inference request in 2.3564910888671875s
Received healthy response to inference request in 2.531313180923462s
2026-03-25T06:03:06.919460+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
Received healthy response to inference request in 2.475403070449829s
Received healthy response to inference request in 2.575754404067993s
Received healthy response to inference request in 2.488757610321045s
Received healthy response to inference request in 2.7290468215942383s
Received healthy response to inference request in 2.4447319507598877s
Received healthy response to inference request in 2.40283465385437s
Received healthy response to inference request in 2.496748208999634s
Received healthy response to inference request in 2.5551061630249023s
Received healthy response to inference request in 2.539843797683716s
Received healthy response to inference request in 2.3725786209106445s
Received healthy response to inference request in 2.522081136703491s
Received healthy response to inference request in 2.506582021713257s
Received healthy response to inference request in 2.7338449954986572s
Received healthy response to inference request in 2.502088785171509s
Received healthy response to inference request in 2.681741952896118s
Received healthy response to inference request in 2.554730176925659s
Received healthy response to inference request in 2.5664749145507812s
Received healthy response to inference request in 2.531137466430664s
Received healthy response to inference request in 2.642223358154297s
Received healthy response to inference request in 2.827592134475708s
Received healthy response to inference request in 2.488844156265259s
Received healthy response to inference request in 2.4475176334381104s
Received healthy response to inference request in 2.5539684295654297s
2026-03-25T06:04:07.133265+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v8
Received healthy response to inference request in 2.7132370471954346s
Received healthy response to inference request in 2.6000797748565674s
30 requests
0 failed requests
5th percentile: 2.363730478286743
10th percentile: 2.3998090505599974
20th percentile: 2.4698259830474854
30th percentile: 2.4943769931793214
40th percentile: 2.5158814907073976
50th percentile: 2.531225323677063
60th percentile: 2.5542731285095215
70th percentile: 2.569258761405945
80th percentile: 2.650127077102661
90th percentile: 2.714818024635315
95th percentile: 2.7316858172416687
99th percentile: 2.8004054641723632
mean time: 2.5461537837982178
Pipeline stage StressChecker completed in 340.72s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.14s
Shutdown handler de-registered
chaiml-qwen35-bobo-19k-_54377_v8 status is now deployed due to DeploymentManager action
chaiml-qwen35-bobo-19k-_54377_v8 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen35-bobo-19k-_54377_v8 status is now torndown due to DeploymentManager action