chaiml-qwen-bobo-19k-rew_1806

developer_uid: rirv938

submission_id: chaiml-qwen-bobo-19k-rew_1806_v2

model_name: chaiml-qwen-bobo-19k-rew_1806_v2

model_group: ChaiML/qwen_bobo_19k_rew

status: torndown

timestamp: 2026-03-27T06:02:13+00:00

num_battles: 9769

num_wins: 4685

celo_rating: 8484.96

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/qwen_bobo_19k_reward_dpo_10k_250_lr1-merged_v

model_architecture: Qwen3_5ForConditionalGeneration

model_num_parameters: 23564784640.0

best_of: 8

max_input_tokens: 2048

max_output_tokens: 80

reward_model: default

display_name: chaiml-qwen-bobo-19k-rew_1806_v2

ineligible_reason: max_output_tokens!=64

is_internal_developer: True

language_model: ChaiML/qwen_bobo_19k_reward_dpo_10k_250_lr1-merged_v

model_size: 24B

ranking_group: single

us_pacific_date: 2026-03-23

win_ratio: 0.47957825775412016

generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '</s>', '<|im_start|>', '<|im_end|>', '###'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}

formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-19k-rew-1806-v2-uploader
Waiting for job on chaiml-qwen-bobo-19k-rew-1806-v2-uploader to finish
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: Using quantization_mode: none
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: Downloading snapshot of ChaiML/qwen_bobo_19k_reward_dpo_10k_250_lr1-merged_v...
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: Downloaded in 32.792s
2026-03-24T03:13:02.084117+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: Processed model ChaiML/qwen_bobo_19k_reward_dpo_10k_250_lr1-merged_v in 52.899s
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-rew-1806-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-19k-rew-1806-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-rew-1806-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-rew-1806-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-rew-1806-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-rew-1806-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-19k-rew-1806-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-19k-rew-1806-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/tokenizer_config.json
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/README.md
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/.gitattributes
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/config.json
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/model.safetensors.index.json
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/generation_config.json
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/tokenizer.json
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/model-00002-of-00002.safetensors
2026-03-24T03:14:02.175540+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
chaiml-qwen-bobo-19k-rew-1806-v2-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-rew-1806-v2/default/model-00001-of-00002.safetensors
Job chaiml-qwen-bobo-19k-rew-1806-v2-uploader completed after 152.86s with status: succeeded
Stopping job with name chaiml-qwen-bobo-19k-rew-1806-v2-uploader
Pipeline stage VLLMUploader completed in 153.32s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.49s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-19k-rew-1806-v2
Waiting for inference service chaiml-qwen-bobo-19k-rew-1806-v2 to be ready
2026-03-24T03:15:02.289353+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
2026-03-24T03:16:02.369168+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
2026-03-24T03:17:02.465605+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
Inference service chaiml-qwen-bobo-19k-rew-1806-v2 ready after 170.43282985687256s
Pipeline stage VLLMDeployer completed in 171.01s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T03:18:02.564616+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T03:19:02.655429+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.481011390686035s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.6071295738220215s
2026-03-24T03:20:02.886733+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
Received healthy response to inference request in 9.24557089805603s
Received healthy response to inference request in 2.869673252105713s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.340924739837646s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-24T03:21:02.982343+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
Received healthy response to inference request in 9.304049491882324s
Received healthy response to inference request in 2.912679433822632s
Received healthy response to inference request in 15.319385051727295s
Received healthy response to inference request in 2.7973175048828125s
Received healthy response to inference request in 2.5217394828796387s
Received healthy response to inference request in 2.597200393676758s
Received healthy response to inference request in 2.9723758697509766s
Received healthy response to inference request in 2.7385621070861816s
Received healthy response to inference request in 2.642918348312378s
Received healthy response to inference request in 2.695674419403076s
Received healthy response to inference request in 2.682508707046509s
Received healthy response to inference request in 2.700573444366455s
Received healthy response to inference request in 2.6914494037628174s
Received healthy response to inference request in 2.6338207721710205s
Received healthy response to inference request in 2.633685350418091s
2026-03-24T03:22:03.072996+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
Received healthy response to inference request in 2.6823360919952393s
30 requests
9 failed requests
5th percentile: 2.6016685247421263
10th percentile: 2.6310297727584837
20th percentile: 2.674452543258667
30th percentile: 2.6944069147109984
40th percentile: 2.7738153457641603
50th percentile: 2.942527651786804
60th percentile: 9.318799591064453
70th percentile: 16.75191977024077
80th percentile: 20.110088062286376
90th percentile: 20.12007212638855
95th percentile: 20.124590051174163
99th percentile: 20.13302891969681
mean time: 9.237095300356547
%s, retrying in %s seconds...
Received healthy response to inference request in 2.5290493965148926s
Received healthy response to inference request in 2.7853505611419678s
Received healthy response to inference request in 2.526081085205078s
Received healthy response to inference request in 2.62864089012146s
Received healthy response to inference request in 2.658538579940796s
Received healthy response to inference request in 2.6818559169769287s
Received healthy response to inference request in 2.706571578979492s
Received healthy response to inference request in 2.5528666973114014s
Received healthy response to inference request in 2.553244113922119s
Received healthy response to inference request in 2.6624720096588135s
Received healthy response to inference request in 2.5676164627075195s
Received healthy response to inference request in 2.8375658988952637s
Received healthy response to inference request in 2.6040823459625244s
Received healthy response to inference request in 2.635063648223877s
Received healthy response to inference request in 2.6714842319488525s
Received healthy response to inference request in 3.1389427185058594s
Received healthy response to inference request in 2.638831853866577s
Received healthy response to inference request in 2.668635606765747s
Received healthy response to inference request in 2.6092312335968018s
Received healthy response to inference request in 2.6054189205169678s
2026-03-24T03:23:03.171860+00:00 monitor updated for chaiml-qwen-bobo-19k-rew_1806_v2
Received healthy response to inference request in 2.9530768394470215s
Received healthy response to inference request in 2.7146799564361572s
Received healthy response to inference request in 2.89007306098938s
Received healthy response to inference request in 2.6312947273254395s
Received healthy response to inference request in 2.627955436706543s
Received healthy response to inference request in 2.6425626277923584s
Received healthy response to inference request in 2.6506712436676025s
Received healthy response to inference request in 2.6695079803466797s
Received healthy response to inference request in 2.8834140300750732s
Received healthy response to inference request in 2.7552716732025146s
30 requests
0 failed requests
5th percentile: 2.5397671818733216
10th percentile: 2.553206372261047
20th percentile: 2.6051516056060793
30th percentile: 2.628435254096985
40th percentile: 2.637324571609497
50th percentile: 2.654604911804199
60th percentile: 2.66898455619812
70th percentile: 2.6892706155776978
80th percentile: 2.7612874507904053
90th percentile: 2.884079933166504
95th percentile: 2.9247251391410827
99th percentile: 3.0850416135787966
mean time: 2.6893350442250568
Pipeline stage StressChecker completed in 362.52s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.66s
Shutdown handler de-registered
chaiml-qwen-bobo-19k-rew_1806_v2 status is now deployed due to DeploymentManager action
chaiml-qwen-bobo-19k-rew_1806_v2 status is now inactive due to admin request
chaiml-qwen-bobo-19k-rew_1806_v2 status is now torndown due to DeploymentManager action