developer_uid: rirv938
submission_id: chaiml-qwen35-bobo-19k-lr3-e3_v4
model_name: chaiml-qwen35-bobo-19k-lr3-e3_v4
model_group: ChaiML/qwen35_bobo_19k_l
status: deployed
timestamp: 2026-03-21T17:24:58+00:00
num_battles: 3550
num_wins: 1696
celo_rating: 1280.72
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen35_bobo_19k_lr3_e3
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 64
reward_model: default
display_name: chaiml-qwen35-bobo-19k-lr3-e3_v4
is_internal_developer: True
language_model: ChaiML/qwen35_bobo_19k_lr3_e3
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-21
win_ratio: 0.47774647887323946
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['###', '<|im_end|>', '<|im_start|>', '</s>', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader
Waiting for job on chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader to finish
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: Using quantization_mode: none
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: Downloading snapshot of ChaiML/qwen35_bobo_19k_lr3_e3...
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: Downloaded in 19.604s
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: Processed model ChaiML/qwen35_bobo_19k_lr3_e3 in 40.992s
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: creating bucket guanaco-vllm-models
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/README.md
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/generation_config.json
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/args.json
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model.safetensors.index.json
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/tokenizer_config.json
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/config.json
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/chat_template.jinja
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/preprocessor_config.json
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/processor_config.json
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/.gitattributes
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/tokenizer.json
2026-03-21T17:16:31.870796+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00001-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00001-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00012-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00012-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00008-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00008-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00010-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00010-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00004-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00004-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00003-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00003-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00005-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00005-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00006-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00006-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00007-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00007-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00002-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00002-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00011-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00011-of-00012.safetensors
chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader: cp /dev/shm/model_output/model-00009-of-00012.safetensors s3://guanaco-vllm-models/chaiml-qwen35-bobo-19k-lr3-e3-v4/default/model-00009-of-00012.safetensors
Job chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader completed after 73.43s with status: succeeded
Stopping job with name chaiml-qwen35-bobo-19k-lr3-e3-v4-uploader
Pipeline stage VLLMUploader completed in 73.89s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.63s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen35-bobo-19k-lr3-e3-v4
Waiting for inference service chaiml-qwen35-bobo-19k-lr3-e3-v4 to be ready
2026-03-21T17:17:31.955825+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
2026-03-21T17:18:32.047287+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
2026-03-21T17:19:32.140994+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
Inference service chaiml-qwen35-bobo-19k-lr3-e3-v4 ready after 180.45578455924988s
Pipeline stage VLLMDeployer completed in 181.02s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-21T17:20:32.230492+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 16.911301136016846s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-21T17:21:32.325545+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.2875027656555176s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-21T17:22:32.417592+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
Received healthy response to inference request in 17.58537983894348s
Received healthy response to inference request in 2.3503522872924805s
Received healthy response to inference request in 2.2203752994537354s
Received healthy response to inference request in 11.781589984893799s
Received healthy response to inference request in 2.263200044631958s
Received healthy response to inference request in 2.184725284576416s
Received healthy response to inference request in 2.1924891471862793s
Received healthy response to inference request in 2.4720137119293213s
Received healthy response to inference request in 2.1730263233184814s
Received healthy response to inference request in 2.154041290283203s
Received healthy response to inference request in 8.89353895187378s
Received healthy response to inference request in 2.167437791824341s
Received healthy response to inference request in 2.3239753246307373s
Received healthy response to inference request in 2.2790021896362305s
2026-03-21T17:23:32.512327+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
Received healthy response to inference request in 2.075928211212158s
Received healthy response to inference request in 2.2259774208068848s
Received healthy response to inference request in 2.2612621784210205s
Received healthy response to inference request in 2.215230941772461s
Received healthy response to inference request in 2.5261497497558594s
Received healthy response to inference request in 2.210399866104126s
Received healthy response to inference request in 2.2621445655822754s
30 requests
7 failed requests
5th percentile: 2.1600697159767153
10th percentile: 2.1724674701690674
20th percentile: 2.2068177223205567
30th percentile: 2.22429678440094
40th percentile: 2.262777853012085
50th percentile: 2.3057390451431274
60th percentile: 2.4936681270599363
70th percentile: 13.320503330230698
80th percentile: 20.113606262207032
90th percentile: 20.126759910583495
95th percentile: 20.131362652778627
99th percentile: 20.13593548297882
mean time: 7.963285112380982
%s, retrying in %s seconds...
Received healthy response to inference request in 2.0433242321014404s
Received healthy response to inference request in 2.07259202003479s
Received healthy response to inference request in 2.1153602600097656s
Received healthy response to inference request in 2.147749423980713s
Received healthy response to inference request in 2.087073564529419s
Received healthy response to inference request in 2.0963070392608643s
Received healthy response to inference request in 2.227407693862915s
Received healthy response to inference request in 2.112354040145874s
Received healthy response to inference request in 2.264829635620117s
Received healthy response to inference request in 2.038728952407837s
Received healthy response to inference request in 2.132553815841675s
Received healthy response to inference request in 2.158574342727661s
Received healthy response to inference request in 2.1621105670928955s
Received healthy response to inference request in 2.1380257606506348s
Received healthy response to inference request in 2.2833492755889893s
Received healthy response to inference request in 2.1771838665008545s
Received healthy response to inference request in 2.1381847858428955s
Received healthy response to inference request in 2.18565034866333s
Received healthy response to inference request in 2.2336997985839844s
2026-03-21T17:24:32.606244+00:00 monitor updated for chaiml-qwen35-bobo-19k-lr3-e3_v4
Received healthy response to inference request in 2.177934169769287s
Received healthy response to inference request in 2.213801145553589s
Received healthy response to inference request in 2.196225166320801s
Received healthy response to inference request in 2.2765491008758545s
Received healthy response to inference request in 2.1669907569885254s
Received healthy response to inference request in 2.194282293319702s
Received healthy response to inference request in 2.1872875690460205s
Received healthy response to inference request in 2.203901529312134s
Received healthy response to inference request in 2.2940869331359863s
Received healthy response to inference request in 2.2815134525299072s
Received healthy response to inference request in 2.462104558944702s
30 requests
0 failed requests
5th percentile: 2.0564947366714477
10th percentile: 2.085625410079956
20th percentile: 2.114759016036987
30th percentile: 2.1381370782852174
40th percentile: 2.1606960773468016
50th percentile: 2.177559018135071
60th percentile: 2.190085458755493
70th percentile: 2.2068714141845702
80th percentile: 2.239925765991211
90th percentile: 2.2816970348358154
95th percentile: 2.2892549872398376
99th percentile: 2.4133794474601746
mean time: 2.1823245366414388
Pipeline stage StressChecker completed in 309.30s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.71s
Shutdown handler de-registered
chaiml-qwen35-bobo-19k-lr3-e3_v4 status is now deployed due to DeploymentManager action