developer_uid: rirv938
submission_id: chaiml-qwen35-bobo-19k-_54377_v2
model_name: chaiml-qwen35-bobo-19k-_54377_v2
model_group: ChaiML/qwen35_bobo_19k-s
status: torndown
timestamp: 2026-03-26T00:56:50+00:00
num_battles: 10236
num_wins: 5565
celo_rating: 1324.31
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen35_bobo_19k-step4455-merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen35-bobo-19k-_54377_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen35_bobo_19k-step4455-merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-22
win_ratio: 0.5436694021101993
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '</s>', '###', '<|im_end|>', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen35-bobo-19k-54377-v2-uploader
Waiting for job on chaiml-qwen35-bobo-19k-54377-v2-uploader to finish
chaiml-qwen35-bobo-19k-54377-v2-uploader: Using quantization_mode: none
chaiml-qwen35-bobo-19k-54377-v2-uploader: Downloading snapshot of ChaiML/qwen35_bobo_19k-step4455-merged...
chaiml-qwen35-bobo-19k-54377-v2-uploader: Downloaded in 18.331s
chaiml-qwen35-bobo-19k-54377-v2-uploader: Processed model ChaiML/qwen35_bobo_19k-step4455-merged in 38.860s
chaiml-qwen35-bobo-19k-54377-v2-uploader: creating bucket guanaco-vllm-models
chaiml-qwen35-bobo-19k-54377-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen35-bobo-19k-54377-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen35-bobo-19k-54377-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen35-bobo-19k-54377-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-54377-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-54377-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen35-bobo-19k-54377-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen35-bobo-19k-54377-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen35-bobo-19k-54377-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen35-bobo-19k-54377-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen35-bobo-19k-54377-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/generation_config.json
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/config.json
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/args.json
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/preprocessor_config.json
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/chat_template.jinja
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/.gitattributes
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/README.md
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model.safetensors.index.json
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/processor_config.json
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/tokenizer_config.json
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/tokenizer.json
2026-03-22T18:02:08.925645+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00001-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00001-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00012-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00012-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00011-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00011-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00009-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00009-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00010-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00010-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00008-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00008-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00004-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00004-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00007-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00007-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00002-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00002-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00005-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00005-of-00012.safetensors
chaiml-qwen35-bobo-19k-54377-v2-uploader: cp /dev/shm/model_output/model-00003-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-54377-v2/default/model-00003-of-00012.safetensors
Job chaiml-qwen35-bobo-19k-54377-v2-uploader completed after 82.72s with status: succeeded
Stopping job with name chaiml-qwen35-bobo-19k-54377-v2-uploader
Pipeline stage VLLMUploader completed in 83.18s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.94s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen35-bobo-19k-54377-v2
Waiting for inference service chaiml-qwen35-bobo-19k-54377-v2 to be ready
2026-03-22T18:03:09.013511+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
2026-03-22T18:04:09.118396+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
2026-03-22T18:05:09.200711+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
Inference service chaiml-qwen35-bobo-19k-54377-v2 ready after 180.41158533096313s
Pipeline stage VLLMDeployer completed in 180.89s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-22T18:06:09.291989+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 17.814216136932373s
2026-03-22T18:07:09.383215+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.4772274494171143s
Received healthy response to inference request in 2.4393718242645264s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.7327544689178467s
Received healthy response to inference request in 9.581547975540161s
Received healthy response to inference request in 9.208271741867065s
2026-03-22T18:08:09.470866+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.576158285140991s
Received healthy response to inference request in 9.5018630027771s
Received healthy response to inference request in 2.557119607925415s
Received healthy response to inference request in 9.871340036392212s
Received healthy response to inference request in 2.5346856117248535s
Received healthy response to inference request in 2.628267526626587s
Received healthy response to inference request in 2.661705493927002s
Received healthy response to inference request in 2.76643967628479s
Received healthy response to inference request in 2.5636260509490967s
Received healthy response to inference request in 2.528812885284424s
Received healthy response to inference request in 2.5626378059387207s
Received healthy response to inference request in 2.6071815490722656s
Received healthy response to inference request in 2.470426082611084s
2026-03-22T18:09:09.782417+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
Received healthy response to inference request in 2.311469078063965s
Received healthy response to inference request in 2.7443466186523438s
Received healthy response to inference request in 2.3824453353881836s
Received healthy response to inference request in 2.6804606914520264s
Received healthy response to inference request in 2.5862069129943848s
30 requests
6 failed requests
5th percentile: 2.408062255382538
10th percentile: 2.4673206567764283
20th percentile: 2.5335110664367675
30th percentile: 2.563329577445984
40th percentile: 2.5987916946411134
50th percentile: 2.671083092689514
60th percentile: 2.7531838417053223
70th percentile: 9.525768494606018
80th percentile: 18.270832061767585
90th percentile: 20.10620586872101
95th percentile: 20.112219655513762
99th percentile: 20.121878566741945
mean time: 7.514660668373108
%s, retrying in %s seconds...
Received healthy response to inference request in 2.7444474697113037s
Received healthy response to inference request in 2.4375791549682617s
Received healthy response to inference request in 2.6082091331481934s
Received healthy response to inference request in 2.603050947189331s
Received healthy response to inference request in 2.460330009460449s
Received healthy response to inference request in 2.708824634552002s
Received healthy response to inference request in 2.466740608215332s
Received healthy response to inference request in 2.4490952491760254s
Received healthy response to inference request in 2.441087245941162s
Received healthy response to inference request in 2.7629947662353516s
Received healthy response to inference request in 2.5282835960388184s
Received healthy response to inference request in 2.3883161544799805s
Received healthy response to inference request in 2.626086473464966s
Received healthy response to inference request in 2.551379919052124s
Received healthy response to inference request in 3.509150505065918s
Received healthy response to inference request in 2.663137674331665s
Received healthy response to inference request in 2.5519561767578125s
2026-03-22T18:10:09.874475+00:00 monitor updated for chaiml-qwen35-bobo-19k-_54377_v2
Received healthy response to inference request in 2.521188974380493s
Received healthy response to inference request in 2.6768712997436523s
Received healthy response to inference request in 2.619582414627075s
Received healthy response to inference request in 2.9453518390655518s
Received healthy response to inference request in 2.5692548751831055s
Received healthy response to inference request in 2.6547622680664062s
Received healthy response to inference request in 2.77668833732605s
Received healthy response to inference request in 2.8510520458221436s
Received healthy response to inference request in 2.4876041412353516s
Received healthy response to inference request in 2.569288730621338s
Received healthy response to inference request in 2.632654905319214s
Received healthy response to inference request in 2.6522810459136963s
Received healthy response to inference request in 2.6858632564544678s
30 requests
0 failed requests
5th percentile: 2.439157795906067
10th percentile: 2.448294448852539
20th percentile: 2.4834314346313477
30th percentile: 2.5444510221481322
40th percentile: 2.5692751884460447
50th percentile: 2.6138957738876343
60th percentile: 2.6405053615570067
70th percentile: 2.6672577619552613
80th percentile: 2.715949201583862
90th percentile: 2.7841247081756593
95th percentile: 2.902916932106018
99th percentile: 3.345648891925812
mean time: 2.638103795051575
Pipeline stage StressChecker completed in 316.99s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.72s
Shutdown handler de-registered
chaiml-qwen35-bobo-19k-_54377_v2 status is now deployed due to DeploymentManager action
chaiml-qwen35-bobo-19k-_54377_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen35-bobo-19k-_54377_v2 status is now torndown due to DeploymentManager action