chaiml-qwen-bobo-reward-d_949

developer_uid: rirv938
submission_id: chaiml-qwen-bobo-reward-d_949_v2
model_name: chaiml-qwen-bobo-reward-d_949_v2
model_group: ChaiML/qwen_bobo_reward_
status: torndown
timestamp: 2026-03-30T07:02:07+00:00
num_battles: 10470
num_wins: 5809
celo_rating: 1333.48
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen_bobo_reward_dpo_user_following_486_merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen-bobo-reward-d_949_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen_bobo_reward_dpo_user_following_486_merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-26
win_ratio: 0.5548233046800382
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', 'You:', '###', '</s>', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-reward-d-949-v2-uploader
Waiting for job on chaiml-qwen-bobo-reward-d-949-v2-uploader to finish
chaiml-qwen-bobo-reward-79090-v3-uploader: Using quantization_mode: none
chaiml-qwen-bobo-reward-79090-v3-uploader: Downloading snapshot of ChaiML/qwen_bobo_reward_dpo_user_plus_fidelity_386_merged...
chaiml-qwen-bobo-reward-d-949-v2-uploader: Using quantization_mode: none
chaiml-qwen-bobo-reward-d-949-v2-uploader: Downloading snapshot of ChaiML/qwen_bobo_reward_dpo_user_following_486_merged...
chaiml-qwen-bobo-reward-d-949-v2-uploader: Downloaded in 31.176s
chaiml-qwen-bobo-reward-79090-v3-uploader: Downloaded in 33.828s
2026-03-27T04:11:36.976473+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
2026-03-27T04:11:40.900805+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
chaiml-qwen-bobo-reward-d-949-v2-uploader: Processed model ChaiML/qwen_bobo_reward_dpo_user_following_486_merged in 51.314s
chaiml-qwen-bobo-reward-d-949-v2-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-reward-d-949-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-d-949-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-reward-d-949-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-reward-d-949-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-reward-d-949-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-d-949-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-reward-d-949-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-d-949-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-reward-d-949-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-d-949-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-reward-d-949-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-d-949-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-reward-d-949-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-reward-d-949-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-reward-d-949-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-reward-d-949-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-reward-d-949-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-reward-d-949-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/tokenizer_config.json
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/README.md
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/model.safetensors.index.json
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/chat_template.jinja
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/generation_config.json
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/config.json
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/.gitattributes
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/tokenizer.json
chaiml-qwen-bobo-reward-79090-v3-uploader: Processed model ChaiML/qwen_bobo_reward_dpo_user_plus_fidelity_386_merged in 54.354s
chaiml-qwen-bobo-reward-79090-v3-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-reward-79090-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-79090-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-reward-79090-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-reward-79090-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-reward-79090-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-79090-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-reward-79090-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-79090-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-reward-79090-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-79090-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-reward-79090-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-reward-79090-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-reward-79090-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-reward-79090-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-reward-79090-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-reward-79090-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-reward-79090-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-reward-79090-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/.gitattributes
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/generation_config.json
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/tokenizer_config.json
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/README.md
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/config.json
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/chat_template.jinja
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/model.safetensors.index.json
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/tokenizer.json
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/model-00002-of-00002.safetensors
chaiml-qwen-bobo-reward-d-949-v2-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-d-949-v2/default/model-00002-of-00002.safetensors
2026-03-27T04:12:37.118388+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
2026-03-27T04:12:41.053289+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
chaiml-qwen-bobo-reward-79090-v3-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-reward-79090-v3/default/model-00001-of-00002.safetensors
Job chaiml-qwen-bobo-reward-79090-v3-uploader completed after 144.5s with status: succeeded
Stopping job with name chaiml-qwen-bobo-reward-79090-v3-uploader
Pipeline stage VLLMUploader completed in 145.12s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.41s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-reward-79090-v3
Waiting for inference service chaiml-qwen-bobo-reward-79090-v3 to be ready
Job chaiml-qwen-bobo-reward-d-949-v2-uploader completed after 144.49s with status: succeeded
Stopping job with name chaiml-qwen-bobo-reward-d-949-v2-uploader
Pipeline stage VLLMUploader completed in 145.21s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.37s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-reward-d-949-v2
Waiting for inference service chaiml-qwen-bobo-reward-d-949-v2 to be ready
2026-03-27T04:13:37.291790+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
2026-03-27T04:13:41.184869+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
2026-03-27T04:14:37.439406+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
2026-03-27T04:14:41.324035+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
Inference service chaiml-qwen-bobo-reward-79090-v3 ready after 150.44843745231628s
Pipeline stage VLLMDeployer completed in 151.23s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-27T04:15:37.577052+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
2026-03-27T04:15:41.484946+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
Inference service chaiml-qwen-bobo-reward-d-949-v2 ready after 160.37275505065918s
Pipeline stage VLLMDeployer completed in 161.10s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T04:16:37.727331+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
2026-03-27T04:16:41.631786+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.565994024276733s
Received healthy response to inference request in 2.516160726547241s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T04:17:37.880482+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
Received healthy response to inference request in 9.695626258850098s
Received healthy response to inference request in 2.4429636001586914s
2026-03-27T04:17:41.818693+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
Received healthy response to inference request in 2.546954393386841s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.655879020690918s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.693702459335327s
Received healthy response to inference request in 9.187586307525635s
Received healthy response to inference request in 2.5021812915802s
Received healthy response to inference request in 9.573094367980957s
Received healthy response to inference request in 2.6416571140289307s
Received healthy response to inference request in 9.428630828857422s
Received healthy response to inference request in 2.5534629821777344s
2026-03-27T04:18:38.043360+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T04:18:42.050874+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
Received healthy response to inference request in 2.981863260269165s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.540102481842041s
Received healthy response to inference request in 2.9921443462371826s
Received healthy response to inference request in 2.578521966934204s
Received healthy response to inference request in 2.6752192974090576s
Received healthy response to inference request in 2.5489249229431152s
Received healthy response to inference request in 2.576575517654419s
Received healthy response to inference request in 2.625748872756958s
Received healthy response to inference request in 3.0142340660095215s
Received healthy response to inference request in 3.471292734146118s
Received healthy response to inference request in 2.612762212753296s
Received healthy response to inference request in 2.6094560623168945s
Received healthy response to inference request in 2.527251958847046s
Received healthy response to inference request in 3.0913374423980713s
Received healthy response to inference request in 2.623809337615967s
Received healthy response to inference request in 2.6912872791290283s
Received healthy response to inference request in 2.6227762699127197s
Received healthy response to inference request in 2.512223720550537s
Received healthy response to inference request in 3.1258387565612793s
Received healthy response to inference request in 2.6341447830200195s
Received healthy response to inference request in 2.6256752014160156s
Received healthy response to inference request in 2.5955584049224854s
Received healthy response to inference request in 2.6461880207061768s
Received healthy response to inference request in 2.7557125091552734s
Received healthy response to inference request in 2.7032108306884766s
Received healthy response to inference request in 2.622894763946533s
Received healthy response to inference request in 2.6249749660491943s
Received healthy response to inference request in 2.8473763465881348s
30 requests
7 failed requests
5th percentile: 2.513995373249054
10th percentile: 2.537708306312561
20th percentile: 2.572602558135986
30th percentile: 2.6188631534576414
40th percentile: 2.6386521816253663
50th percentile: 2.7247074842453003
60th percentile: 3.0256529331207274
70th percentile: 9.568124127388
80th percentile: 20.249257373809815
Received healthy response to inference request in 2.798658847808838s
90th percentile: 20.282651662826538
95th percentile: 20.289342510700227
99th percentile: 20.308451147079467
mean time: 7.490898187955221
%s, retrying in %s seconds...
Received healthy response to inference request in 2.5949037075042725s
Received healthy response to inference request in 2.4917187690734863s
Received healthy response to inference request in 2.5928218364715576s
Received healthy response to inference request in 2.543649435043335s
Received healthy response to inference request in 2.732344388961792s
Received healthy response to inference request in 2.514922618865967s
Received healthy response to inference request in 2.8495657444000244s
Received healthy response to inference request in 2.608877182006836s
30 requests
7 failed requests
5th percentile: 2.539046919345856
10th percentile: 2.5742642641067506
20th percentile: 2.609190511703491
30th percentile: 2.6246252775192263
40th percentile: 2.663606786727905
50th percentile: 2.765501618385315
60th percentile: 3.0009802341461183
2026-03-27T04:19:38.237303+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
70th percentile: 9.25989966392517
80th percentile: 20.243327951431276
90th percentile: 20.27400040626526
95th percentile: 20.282944571971893
99th percentile: 20.289593386650086
Received healthy response to inference request in 2.765286922454834s
mean time: 7.47212963104248
%s, retrying in %s seconds...
Received healthy response to inference request in 2.4818263053894043s
2026-03-27T04:19:42.236430+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
Received healthy response to inference request in 2.6552584171295166s
Received healthy response to inference request in 2.4452860355377197s
Received healthy response to inference request in 2.432535171508789s
Received healthy response to inference request in 2.4347357749938965s
Received healthy response to inference request in 2.428966760635376s
Received healthy response to inference request in 2.7013986110687256s
Received healthy response to inference request in 2.981163740158081s
Received healthy response to inference request in 2.5392584800720215s
Received healthy response to inference request in 9.324707746505737s
Received healthy response to inference request in 2.511159658432007s
Received healthy response to inference request in 2.5068697929382324s
Received healthy response to inference request in 2.6803958415985107s
Received healthy response to inference request in 2.686236619949341s
Received healthy response to inference request in 2.4988651275634766s
Received healthy response to inference request in 2.5257372856140137s
Received healthy response to inference request in 2.516659736633301s
Received healthy response to inference request in 2.588348627090454s
Received healthy response to inference request in 2.513739824295044s
Received healthy response to inference request in 2.535158634185791s
Received healthy response to inference request in 2.5179080963134766s
Received healthy response to inference request in 2.560413122177124s
Received healthy response to inference request in 2.5620105266571045s
Received healthy response to inference request in 2.4914090633392334s
Received healthy response to inference request in 2.776426315307617s
Received healthy response to inference request in 2.7489449977874756s
Received healthy response to inference request in 2.6851844787597656s
Received healthy response to inference request in 2.602766752243042s
Received healthy response to inference request in 2.6419930458068848s
Received healthy response to inference request in 2.7060985565185547s
Received healthy response to inference request in 2.476374864578247s
Received healthy response to inference request in 2.624343156814575s
Received healthy response to inference request in 2.5775840282440186s
Received healthy response to inference request in 2.6511709690093994s
Received healthy response to inference request in 2.839838743209839s
Received healthy response to inference request in 2.636734962463379s
Received healthy response to inference request in 2.6082355976104736s
Received healthy response to inference request in 2.5750725269317627s
Received healthy response to inference request in 2.8082778453826904s
Received healthy response to inference request in 2.648408889770508s
2026-03-27T04:20:38.671432+00:00 monitor updated for chaiml-qwen-bobo-reward_79090_v3
Received healthy response to inference request in 2.678896188735962s
Received healthy response to inference request in 2.566908836364746s
2026-03-27T04:20:42.466416+00:00 monitor updated for chaiml-qwen-bobo-reward-d_949_v2
Received healthy response to inference request in 2.630917549133301s
Received healthy response to inference request in 2.594960927963257s
Received healthy response to inference request in 2.5926430225372314s
Received healthy response to inference request in 2.581766366958618s
Received healthy response to inference request in 2.662419319152832s
Received healthy response to inference request in 2.6350181102752686s
Received healthy response to inference request in 2.6673924922943115s
Received healthy response to inference request in 2.482020616531372s
Received healthy response to inference request in 2.577582359313965s
Received healthy response to inference request in 2.623554229736328s
30 requests
0 failed requests
5th percentile: 2.4617291569709776
10th percentile: 2.4820011854171753
20th percentile: 2.5038395881652833
30th percentile: 2.5323322296142576
40th percentile: 2.5643105506896973
50th percentile: 2.585057497024536
60th percentile: 2.6052109241485595
70th percentile: 2.627545642852783
80th percentile: 2.648961305618286
Received healthy response to inference request in 2.688328981399536s
90th percentile: 2.7103832006454467
95th percentile: 2.757933056354523
99th percentile: 7.422475707530981
mean time: 2.806098461151123
Pipeline stage StressChecker completed in 323.54s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.16s
Shutdown handler de-registered
chaiml-qwen-bobo-reward_79090_v3 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 2.8091964721679688s
Received healthy response to inference request in 2.7352542877197266s
30 requests
0 failed requests
5th percentile: 2.4522630333900453
10th percentile: 2.4966161012649537
20th percentile: 2.5160757541656493
30th percentile: 2.5551849126815798
40th percentile: 2.586619424819946
50th percentile: 2.6364552974700928
60th percentile: 2.6644085884094237
70th percentile: 2.681832432746887
80th percentile: 2.708169746398926
90th percentile: 2.8083697080612184
95th percentile: 2.8260497212409974
99th percentile: 2.9401794910430907
mean time: 2.6331955194473267
Pipeline stage StressChecker completed in 315.72s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.73s
Shutdown handler de-registered
chaiml-qwen-bobo-reward-d_949_v2 status is now deployed due to DeploymentManager action
chaiml-qwen-bobo-reward-d_949_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen-bobo-reward-d_949_v2 status is now torndown due to DeploymentManager action