chaiml-qwen-bobo-19k-re_33926

developer_uid: chai_backend_admin

submission_id: chaiml-qwen-bobo-19k-re_33926_v3

model_name: chaiml-qwen-bobo-19k-re_33926_v3

model_group: ChaiML/qwen_bobo_19k_rew

status: torndown

timestamp: 2026-03-28T22:41:45+00:00

num_battles: 11366

num_wins: 6384

celo_rating: 1329.55

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/qwen_bobo_19k_reward_dpo_10k_250_higher_dpo_2_merged

model_architecture: Qwen3_5ForConditionalGeneration

model_num_parameters: 23564784640.0

best_of: 8

max_input_tokens: 2048

max_output_tokens: 80

reward_model: default

display_name: chaiml-qwen-bobo-19k-re_33926_v3

ineligible_reason: max_output_tokens!=64

is_internal_developer: True

language_model: ChaiML/qwen_bobo_19k_reward_dpo_10k_250_higher_dpo_2_merged

model_size: 24B

ranking_group: single

us_pacific_date: 2026-03-25

win_ratio: 0.5616751715643146

generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['###', 'You:', '<|im_end|>', '<|im_start|>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}

formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-19k-re-33926-v3-uploader
Waiting for job on chaiml-qwen-bobo-19k-re-33926-v3-uploader to finish
chaiml-qwen-bobo-19k-re-33926-v3-uploader: Using quantization_mode: none
chaiml-qwen-bobo-19k-re-33926-v3-uploader: Downloading snapshot of ChaiML/qwen_bobo_19k_reward_dpo_10k_250_higher_dpo_2_merged...
chaiml-qwen-bobo-19k-re-33926-v3-uploader: Downloaded in 31.349s
2026-03-25T20:44:00.797067+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
chaiml-qwen-bobo-19k-re-33926-v3-uploader: Processed model ChaiML/qwen_bobo_19k_reward_dpo_10k_250_higher_dpo_2_merged in 52.413s
chaiml-qwen-bobo-19k-re-33926-v3-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-19k-re-33926-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-19k-re-33926-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-19k-re-33926-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-19k-re-33926-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-19k-re-33926-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-19k-re-33926-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-19k-re-33926-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-19k-re-33926-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-19k-re-33926-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-19k-re-33926-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-19k-re-33926-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-19k-re-33926-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-19k-re-33926-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/.gitattributes
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/README.md
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/chat_template.jinja
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/generation_config.json
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/tokenizer_config.json
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/model.safetensors.index.json
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/config.json
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/tokenizer.json
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/model-00002-of-00002.safetensors
2026-03-25T20:45:00.979221+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
chaiml-qwen-bobo-19k-re-33926-v3-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v3/default/model-00001-of-00002.safetensors
Job chaiml-qwen-bobo-19k-re-33926-v3-uploader completed after 143.2s with status: succeeded
Stopping job with name chaiml-qwen-bobo-19k-re-33926-v3-uploader
Pipeline stage VLLMUploader completed in 143.70s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.12s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-19k-re-33926-v3
Waiting for inference service chaiml-qwen-bobo-19k-re-33926-v3 to be ready
2026-03-25T20:46:01.063039+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
2026-03-25T20:47:01.151200+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
2026-03-25T20:48:01.238336+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
Inference service chaiml-qwen-bobo-19k-re-33926-v3 ready after 170.48469042778015s
Pipeline stage VLLMDeployer completed in 171.05s
run pipeline stage %s
Running pipeline stage StressChecker
Failed to get request counts for guanaco-submitter. Falling back to default
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T20:49:01.347067+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 10.492415428161621s
2026-03-25T20:50:01.436528+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 10.569276809692383s
Received healthy response to inference request in 3.7284271717071533s
Received healthy response to inference request in 3.758988857269287s
Received healthy response to inference request in 4.0707457065582275s
2026-03-25T20:51:01.525652+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
Received healthy response to inference request in 18.051940202713013s
Received healthy response to inference request in 10.972561597824097s
Received healthy response to inference request in 3.9167208671569824s
Received healthy response to inference request in 3.8937010765075684s
Received healthy response to inference request in 4.162353754043579s
Received healthy response to inference request in 3.988828659057617s
Received healthy response to inference request in 3.882951021194458s
Received healthy response to inference request in 4.596938610076904s
Received healthy response to inference request in 3.9832839965820312s
Received healthy response to inference request in 4.473917245864868s
Received healthy response to inference request in 3.9312801361083984s
2026-03-25T20:52:01.616634+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
Received healthy response to inference request in 4.04181694984436s
Received healthy response to inference request in 3.8866171836853027s
Received healthy response to inference request in 4.019479513168335s
Received healthy response to inference request in 3.9147982597351074s
Received healthy response to inference request in 3.982827663421631s
Received healthy response to inference request in 4.161390542984009s
Received healthy response to inference request in 4.000195741653442s
Received healthy response to inference request in 3.9267454147338867s
30 requests
6 failed requests
5th percentile: 3.814771831035614
10th percentile: 3.8862505674362184
20th percentile: 3.9163363456726072
30th percentile: 3.967363405227661
40th percentile: 3.995648908615112
50th percentile: 4.056281328201294
60th percentile: 4.286979150772094
70th percentile: 10.51547384262085
80th percentile: 18.463555955886847
90th percentile: 20.122656846046446
95th percentile: 20.131862235069274
99th percentile: 20.152632217407227
mean time: 8.372839291890463
%s, retrying in %s seconds...
Received healthy response to inference request in 3.7003304958343506s
Received healthy response to inference request in 3.7516117095947266s
Received healthy response to inference request in 3.7507314682006836s
Received healthy response to inference request in 3.7820844650268555s
Received healthy response to inference request in 3.8157100677490234s
Received healthy response to inference request in 3.80230712890625s
Received healthy response to inference request in 3.7951831817626953s
2026-03-25T20:53:01.817892+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
Received healthy response to inference request in 4.413109064102173s
Received healthy response to inference request in 3.8115551471710205s
Received healthy response to inference request in 3.9582653045654297s
Received healthy response to inference request in 4.43418025970459s
Received healthy response to inference request in 3.8604142665863037s
Received healthy response to inference request in 3.827255964279175s
Received healthy response to inference request in 3.942537784576416s
Received healthy response to inference request in 4.076748371124268s
Failed to get response for submission qwen-qwen3-5-35b-a3b-fp8_v10: ('http://qwen-qwen3-5-35b-a3b-fp8-v10-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'request timeout')
Received healthy response to inference request in 3.8182783126831055s
Received healthy response to inference request in 3.963750123977661s
Received healthy response to inference request in 3.98626971244812s
Received healthy response to inference request in 3.930966377258301s
Received healthy response to inference request in 4.0956902503967285s
Received healthy response to inference request in 3.980156183242798s
Received healthy response to inference request in 4.091367721557617s
2026-03-25T20:54:01.918684+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v3
Received healthy response to inference request in 3.941204786300659s
Received healthy response to inference request in 3.937011241912842s
Received healthy response to inference request in 3.984502077102661s
Received healthy response to inference request in 4.140132427215576s
Received healthy response to inference request in 4.113421440124512s
Received healthy response to inference request in 3.9951560497283936s
Received healthy response to inference request in 4.263997554779053s
Received healthy response to inference request in 4.179121971130371s
30 requests
0 failed requests
5th percentile: 3.751127576828003
10th percentile: 3.7790371894836428
20th percentile: 3.8097055435180662
30th percentile: 3.824562668800354
40th percentile: 3.9345932960510255
50th percentile: 3.950401544570923
60th percentile: 3.981894540786743
70th percentile: 4.019633746147155
80th percentile: 4.099236488342285
90th percentile: 4.18760952949524
95th percentile: 4.346008884906769
99th percentile: 4.428069612979889
mean time: 3.971435030301412
Pipeline stage StressChecker completed in 376.38s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.12s
Shutdown handler de-registered
chaiml-qwen-bobo-19k-re_33926_v3 status is now deployed due to DeploymentManager action
chaiml-qwen-bobo-19k-re_33926_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen-bobo-19k-re_33926_v3 status is now torndown due to DeploymentManager action