developer_uid: rirv938
submission_id: chaiml-qwen-bobo-dpo-ju_40611_v2
model_name: chaiml-qwen-bobo-dpo-ju_40611_v2
model_group: ChaiML/qwen_bobo_dpo_jud
status: torndown
timestamp: 2026-04-07T20:51:05+00:00
num_battles: 11902
num_wins: 6552
celo_rating: 1330.83
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen_bobo_dpo_judge_plus_reward_700_much_safer_merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen-bobo-dpo-ju_40611_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen_bobo_dpo_judge_plus_reward_700_much_safer_merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5504957150058813
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_end|>', '###', '<|im_start|>', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-dpo-ju-40611-v2-uploader
Waiting for job on chaiml-qwen-bobo-dpo-ju-40611-v2-uploader to finish
chaiml-merged-qwen-35-39140-v21-uploader: Using quantization_mode: none
chaiml-merged-qwen-35-39140-v21-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v...
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: Using quantization_mode: none
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: Downloading snapshot of ChaiML/qwen_bobo_dpo_judge_plus_reward_700_much_safer_merged...
chaiml-merged-qwen-35-39140-v21-uploader: Downloaded in 32.038s
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: Downloaded in 30.718s
2026-03-28T21:02:49.566118+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
chaiml-merged-qwen-35-39140-v21-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 52.216s
chaiml-merged-qwen-35-39140-v21-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-39140-v21-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v21-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-39140-v21-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-39140-v21-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-39140-v21-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v21-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v21-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v21-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v21-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v21-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v21-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v21-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v21-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-39140-v21-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-39140-v21-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-39140-v21-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-39140-v21-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-merged-qwen-35-39140-v21-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/.gitattributes
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/README.md
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/generation_config.json
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/model.safetensors.index.json
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/config.json
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/tokenizer_config.json
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/tokenizer.json
2026-03-28T21:02:53.571153+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: Processed model ChaiML/qwen_bobo_dpo_judge_plus_reward_700_much_safer_merged in 51.652s
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/README.md
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/model.safetensors.index.json
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/chat_template.jinja
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/generation_config.json
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/.gitattributes
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/tokenizer_config.json
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/config.json
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/tokenizer.json
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/model-00002-of-00002.safetensors
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/model-00002-of-00002.safetensors
2026-03-28T21:03:49.702732+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
2026-03-28T21:03:53.702368+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
chaiml-merged-qwen-35-39140-v21-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v21/default/model-00001-of-00002.safetensors
Job chaiml-merged-qwen-35-39140-v21-uploader completed after 144.01s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-39140-v21-uploader
Pipeline stage VLLMUploader completed in 144.59s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.40s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-merged-qwen-35-39140-v21
Waiting for inference service chaiml-merged-qwen-35-39140-v21 to be ready
chaiml-qwen-bobo-dpo-ju-40611-v2-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-40611-v2/default/model-00001-of-00002.safetensors
Job chaiml-qwen-bobo-dpo-ju-40611-v2-uploader completed after 144.24s with status: succeeded
Stopping job with name chaiml-qwen-bobo-dpo-ju-40611-v2-uploader
Pipeline stage VLLMUploader completed in 144.93s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.32s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-dpo-ju-40611-v2
Waiting for inference service chaiml-qwen-bobo-dpo-ju-40611-v2 to be ready
2026-03-28T21:04:49.837840+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
2026-03-28T21:04:53.835604+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
2026-03-28T21:05:49.983869+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
2026-03-28T21:05:53.981521+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
2026-03-28T21:06:50.115097+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
2026-03-28T21:06:54.115345+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
Inference service chaiml-qwen-bobo-dpo-ju-40611-v2 ready after 160.18145942687988s
Pipeline stage VLLMDeployer completed in 160.74s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service chaiml-merged-qwen-35-39140-v21 ready after 170.55587196350098s
Pipeline stage VLLMDeployer completed in 171.18s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T21:07:50.258024+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
2026-03-28T21:07:54.249358+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T21:08:50.414604+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
2026-03-28T21:08:54.394002+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
Received healthy response to inference request in 9.580842733383179s
Received healthy response to inference request in 16.081773281097412s
Received healthy response to inference request in 3.3974761962890625s
Received healthy response to inference request in 9.276586532592773s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.4179821014404297s
Received healthy response to inference request in 2.3986759185791016s
2026-03-28T21:09:50.565053+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.442643642425537s
Received healthy response to inference request in 2.5919787883758545s
2026-03-28T21:09:54.545816+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
Received healthy response to inference request in 2.243185520172119s
Received healthy response to inference request in 2.440208673477173s
Received healthy response to inference request in 2.473932981491089s
Received healthy response to inference request in 9.110939979553223s
Received healthy response to inference request in 2.609562873840332s
Received healthy response to inference request in 2.433560848236084s
Received healthy response to inference request in 2.415825843811035s
Received healthy response to inference request in 2.9185948371887207s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.63259220123291s
Received healthy response to inference request in 2.4570319652557373s
Received healthy response to inference request in 12.741575241088867s
Received healthy response to inference request in 2.6178274154663086s
Received healthy response to inference request in 9.188092231750488s
2026-03-28T21:10:50.727346+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
Received healthy response to inference request in 2.8376083374023438s
Received healthy response to inference request in 2.465796947479248s
2026-03-28T21:10:54.711632+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
Received healthy response to inference request in 2.4131863117218018s
Received healthy response to inference request in 2.5763139724731445s
Received healthy response to inference request in 2.450394868850708s
Received healthy response to inference request in 2.61541748046875s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.4660913944244385s
Received healthy response to inference request in 2.7586281299591064s
Received healthy response to inference request in 2.4395761489868164s
Received healthy response to inference request in 2.3673667907714844s
Received healthy response to inference request in 2.4991230964660645s
Received healthy response to inference request in 2.194228410720825s
Received healthy response to inference request in 2.4678146839141846s
30 requests
8 failed requests
5th percentile: 2.423806595802307
10th percentile: 2.438974618911743
20th percentile: 2.464043951034546
30th percentile: 2.4897305727005006
40th percentile: 2.602529239654541
50th percentile: 2.735100269317627
60th percentile: 5.682861709594718
70th percentile: 11.318142557144146
80th percentile: 20.239211225509642
90th percentile: 20.25247142314911
95th percentile: 20.25488015413284
Received healthy response to inference request in 2.428673028945923s
99th percentile: 20.308215427398682
mean time: 8.41310526529948
%s, retrying in %s seconds...
Received healthy response to inference request in 2.47735333442688s
Received healthy response to inference request in 2.5274016857147217s
Received healthy response to inference request in 2.583853244781494s
Received healthy response to inference request in 2.657496690750122s
Received healthy response to inference request in 2.53969144821167s
Received healthy response to inference request in 2.392685890197754s
Received healthy response to inference request in 2.54878830909729s
Received healthy response to inference request in 9.126548290252686s
Received healthy response to inference request in 2.5025713443756104s
Received healthy response to inference request in 2.5274415016174316s
Received healthy response to inference request in 2.419203519821167s
Received healthy response to inference request in 2.4514782428741455s
Received healthy response to inference request in 2.4885826110839844s
Received healthy response to inference request in 2.4376766681671143s
Received healthy response to inference request in 2.491597890853882s
Received healthy response to inference request in 2.4248709678649902s
Received healthy response to inference request in 2.5014679431915283s
Received healthy response to inference request in 2.7252097129821777s
30 requests
9 failed requests
5th percentile: 2.2990670919418337
10th percentile: 2.39554500579834
20th percentile: 2.4279126167297362
30th percentile: 2.4480973720550536
40th percentile: 2.507406234741211
50th percentile: 2.671518564224243
60th percentile: 9.252986431121826
Received healthy response to inference request in 2.415438652038574s
70th percentile: 14.986024713516214
80th percentile: 20.24092879295349
90th percentile: 20.24403295516968
95th percentile: 20.26488342285156
99th percentile: 28.03340855836869
mean time: 9.200659720102946
%s, retrying in %s seconds...
Received healthy response to inference request in 2.4880855083465576s
Received healthy response to inference request in 2.4372425079345703s
2026-03-28T21:11:50.894147+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v21
Received healthy response to inference request in 2.452634334564209s
Received healthy response to inference request in 2.357863664627075s
Received healthy response to inference request in 2.657904863357544s
2026-03-28T21:11:54.906452+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_40611_v2
Received healthy response to inference request in 2.270787239074707s
Received healthy response to inference request in 2.5453569889068604s
Received healthy response to inference request in 2.4275007247924805s
Received healthy response to inference request in 2.6178715229034424s
Received healthy response to inference request in 2.392723560333252s
Received healthy response to inference request in 2.4428908824920654s
Received healthy response to inference request in 2.3667852878570557s
Received healthy response to inference request in 2.424081325531006s
Received healthy response to inference request in 2.3980660438537598s
Received healthy response to inference request in 2.5395376682281494s
Received healthy response to inference request in 2.4074764251708984s
Received healthy response to inference request in 2.4595909118652344s
Received healthy response to inference request in 2.473527431488037s
Received healthy response to inference request in 2.4409687519073486s
Received healthy response to inference request in 2.389216899871826s
Received healthy response to inference request in 2.4049086570739746s
Received healthy response to inference request in 2.7409777641296387s
Received healthy response to inference request in 2.4098074436187744s
Received healthy response to inference request in 2.451355457305908s
Received healthy response to inference request in 2.5918970108032227s
Received healthy response to inference request in 2.4522080421447754s
Received healthy response to inference request in 2.475623607635498s
Received healthy response to inference request in 2.631333589553833s
Received healthy response to inference request in 2.462933301925659s
Received healthy response to inference request in 2.595372200012207s
Received healthy response to inference request in 2.502885341644287s
Received healthy response to inference request in 2.741448402404785s
Received healthy response to inference request in 2.466110944747925s
Received healthy response to inference request in 2.5543951988220215s
Received healthy response to inference request in 2.5175037384033203s
Received healthy response to inference request in 2.4491841793060303s
Received healthy response to inference request in 2.416463851928711s
30 requests
0 failed requests
Received healthy response to inference request in 2.649240732192993s
5th percentile: 2.4158999919891357
10th percentile: 2.4189295530319215
20th percentile: 2.442506456375122
30th percentile: 2.452506446838379
40th percentile: 2.471818542480469
50th percentile: 2.490090250968933
60th percentile: 2.5026969432830812
70th percentile: 2.53104248046875
80th percentile: 2.5460432529449464
90th percentile: 2.6218340396881104
95th percentile: 2.657721185684204
99th percentile: 2.716886622905731
mean time: 2.505744989713033
Pipeline stage StressChecker completed in 339.65s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Received healthy response to inference request in 2.475057601928711s
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.13s
Shutdown handler de-registered
chaiml-qwen-bobo-dpo-ju_40611_v2 status is now deployed due to DeploymentManager action
chaiml-qwen-bobo-dpo-ju_40611_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen-bobo-dpo-ju_40611_v2 status is now torndown due to DeploymentManager action