chaiml-qwen-bobo-19k-re_33926

developer_uid: rirv938

submission_id: chaiml-qwen-bobo-19k-re_33926_v1

model_name: chaiml-qwen-bobo-19k-re_33926_v1

model_group: ChaiML/qwen_bobo_19k_rew

status: torndown

timestamp: 2026-03-28T15:18:54+00:00

num_battles: 10332

num_wins: 5894

celo_rating: 1342.37

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/qwen_bobo_19k_reward_dpo_10k_250_higher_dpo_2_merged

model_architecture: Qwen3_5ForConditionalGeneration

model_num_parameters: 23564784640.0

best_of: 8

max_input_tokens: 2048

max_output_tokens: 80

reward_model: default

display_name: chaiml-qwen-bobo-19k-re_33926_v1

ineligible_reason: max_output_tokens!=64

is_internal_developer: True

language_model: ChaiML/qwen_bobo_19k_reward_dpo_10k_250_higher_dpo_2_merged

model_size: 24B

ranking_group: single

us_pacific_date: 2026-03-25

win_ratio: 0.5704607046070461

generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '</s>', '###', '<|im_start|>', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}

formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-19k-re-33926-v1-uploader
Waiting for job on chaiml-qwen-bobo-19k-re-33926-v1-uploader to finish
chaiml-qwen-bobo-19k-re-33926-v1-uploader: Using quantization_mode: none
chaiml-qwen-bobo-19k-re-33926-v1-uploader: Downloading snapshot of ChaiML/qwen_bobo_19k_reward_dpo_10k_250_higher_dpo_2_merged...
chaiml-qwen-bobo-19k-re-33926-v1-uploader: Downloaded in 31.493s
2026-03-25T05:55:10.051794+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
chaiml-qwen-bobo-19k-re-33926-v1-uploader: Processed model ChaiML/qwen_bobo_19k_reward_dpo_10k_250_higher_dpo_2_merged in 52.035s
chaiml-qwen-bobo-19k-re-33926-v1-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-19k-re-33926-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-19k-re-33926-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-19k-re-33926-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-19k-re-33926-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-19k-re-33926-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-19k-re-33926-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-19k-re-33926-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-19k-re-33926-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-19k-re-33926-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-19k-re-33926-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-19k-re-33926-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-19k-re-33926-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-19k-re-33926-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-19k-re-33926-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/.gitattributes
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/model.safetensors.index.json
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/generation_config.json
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/chat_template.jinja
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/README.md
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/config.json
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/tokenizer_config.json
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/tokenizer.json
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/model-00002-of-00002.safetensors
2026-03-25T05:56:10.143019+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
chaiml-qwen-bobo-19k-re-33926-v1-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-19k-re-33926-v1/default/model-00001-of-00002.safetensors
Job chaiml-qwen-bobo-19k-re-33926-v1-uploader completed after 147.67s with status: succeeded
Stopping job with name chaiml-qwen-bobo-19k-re-33926-v1-uploader
Pipeline stage VLLMUploader completed in 148.14s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.83s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-19k-re-33926-v1
Waiting for inference service chaiml-qwen-bobo-19k-re-33926-v1 to be ready
2026-03-25T05:57:10.233881+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
2026-03-25T05:58:10.330724+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
2026-03-25T05:59:10.436763+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
Inference service chaiml-qwen-bobo-19k-re-33926-v1 ready after 180.41724491119385s
Pipeline stage VLLMDeployer completed in 180.87s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T06:00:10.528824+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T06:01:10.621533+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.551815748214722s
2026-03-25T06:02:10.715665+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.541522979736328s
Received healthy response to inference request in 9.131255626678467s
Received healthy response to inference request in 2.764207124710083s
Received healthy response to inference request in 2.5799248218536377s
2026-03-25T06:03:10.809406+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.9705705642700195s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T06:04:10.912907+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
Received healthy response to inference request in 14.01513934135437s
Received healthy response to inference request in 2.7682299613952637s
Received healthy response to inference request in 2.8611197471618652s
Received healthy response to inference request in 2.8879165649414062s
Received healthy response to inference request in 2.751436710357666s
Received healthy response to inference request in 2.561070442199707s
Received healthy response to inference request in 2.5420212745666504s
Received healthy response to inference request in 2.5935816764831543s
Received healthy response to inference request in 2.746455192565918s
Received healthy response to inference request in 2.637331008911133s
Received healthy response to inference request in 2.567908525466919s
Received healthy response to inference request in 2.5954091548919678s
Received healthy response to inference request in 2.6875741481781006s
30 requests
11 failed requests
5th percentile: 2.564147579669952
10th percentile: 2.578723192214966
20th percentile: 2.6289466381073
30th percentile: 2.749942255020142
40th percentile: 2.823963832855225
50th percentile: 6.050913095474243
60th percentile: 11.337145185470575
70th percentile: 20.11023645401001
80th percentile: 20.122120475769044
90th percentile: 20.14808428287506
95th percentile: 20.158552503585817
99th percentile: 20.164096641540528
mean time: 10.140248330434163
%s, retrying in %s seconds...
Received healthy response to inference request in 2.4615936279296875s
Received healthy response to inference request in 2.611490249633789s
Received healthy response to inference request in 2.479339599609375s
Received healthy response to inference request in 2.5252110958099365s
Received healthy response to inference request in 2.7352447509765625s
Received healthy response to inference request in 2.65173602104187s
Received healthy response to inference request in 2.4995510578155518s
Received healthy response to inference request in 2.5113444328308105s
Received healthy response to inference request in 2.4906013011932373s
2026-03-25T06:05:11.013311+00:00 monitor updated for chaiml-qwen-bobo-19k-re_33926_v1
Received healthy response to inference request in 2.530259370803833s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.8526108264923096s
Received healthy response to inference request in 2.3814539909362793s
Received healthy response to inference request in 2.5603957176208496s
chaiml-qwen-bobo-19k-re_33926_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen-bobo-19k-re_33926_v1 status is now torndown due to DeploymentManager action