chaiml-qwen-bobo-dpo-ju_56781

developer_uid: rirv938
submission_id: chaiml-qwen-bobo-dpo-ju_56781_v1
model_name: chaiml-qwen-bobo-dpo-ju_56781_v1
model_group: ChaiML/qwen_bobo_dpo_jud
status: torndown
timestamp: 2026-03-31T09:01:40+00:00
num_battles: 10193
num_wins: 5794
celo_rating: 1342.0
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen-bobo-dpo-ju_56781_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5684293142352594
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_start|>', '###', '<|im_end|>', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-dpo-ju-56781-v1-uploader
Waiting for job on chaiml-qwen-bobo-dpo-ju-56781-v1-uploader to finish
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: Using quantization_mode: none
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: Downloading snapshot of ChaiML/qwen_bobo_dpo_judge_plus_reward_750_merged...
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: Using quantization_mode: none
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: Downloading snapshot of ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged...
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: Downloaded in 30.337s
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: Downloaded in 36.263s
2026-03-27T23:44:27.146208+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
2026-03-27T23:44:30.979598+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: Processed model ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged in 50.558s
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/.gitattributes
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/generation_config.json
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/model.safetensors.index.json
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/README.md
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/config.json
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/chat_template.jinja
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/tokenizer_config.json
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/tokenizer.json
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: Processed model ChaiML/qwen_bobo_dpo_judge_plus_reward_750_merged in 56.117s
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/README.md
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/generation_config.json
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/tokenizer_config.json
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/chat_template.jinja
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/config.json
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/model.safetensors.index.json
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/.gitattributes
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/tokenizer.json
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/model-00002-of-00002.safetensors
2026-03-27T23:45:27.276818+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
2026-03-27T23:45:31.108979+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
chaiml-qwen-bobo-dpo-ju-56781-v1-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v1/default/model-00001-of-00002.safetensors
Job chaiml-qwen-bobo-dpo-ju-56781-v1-uploader completed after 154.25s with status: succeeded
Stopping job with name chaiml-qwen-bobo-dpo-ju-56781-v1-uploader
Pipeline stage VLLMUploader completed in 154.95s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.82s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-dpo-ju-56781-v1
Waiting for inference service chaiml-qwen-bobo-dpo-ju-56781-v1 to be ready
chaiml-qwen-bobo-dpo-ju-49011-v2-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-49011-v2/default/model-00001-of-00002.safetensors
Job chaiml-qwen-bobo-dpo-ju-49011-v2-uploader completed after 165.1s with status: succeeded
Stopping job with name chaiml-qwen-bobo-dpo-ju-49011-v2-uploader
Pipeline stage VLLMUploader completed in 165.67s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.38s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-dpo-ju-49011-v2
Waiting for inference service chaiml-qwen-bobo-dpo-ju-49011-v2 to be ready
2026-03-27T23:46:27.411463+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
2026-03-27T23:46:31.251103+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
2026-03-27T23:47:27.594041+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
2026-03-27T23:47:31.399330+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
2026-03-27T23:48:27.736313+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
2026-03-27T23:48:31.548009+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
Inference service chaiml-qwen-bobo-dpo-ju-56781-v1 ready after 170.4223244190216s
Pipeline stage VLLMDeployer completed in 171.16s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service chaiml-qwen-bobo-dpo-ju-49011-v2 ready after 170.45482110977173s
Pipeline stage VLLMDeployer completed in 171.56s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T23:49:27.925080+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
2026-03-27T23:49:31.692817+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-27T23:50:28.091013+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
2026-03-27T23:50:31.844879+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
Received healthy response to inference request in 16.32452392578125s
Received healthy response to inference request in 2.5163872241973877s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 18.290469884872437s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.638415098190308s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.358940124511719s
2026-03-27T23:51:28.269879+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
Received healthy response to inference request in 9.7633216381073s
2026-03-27T23:51:32.771970+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.8445217609405518s
Received healthy response to inference request in 2.7870254516601562s
Received healthy response to inference request in 2.4335904121398926s
Received healthy response to inference request in 3.3155617713928223s
Received healthy response to inference request in 2.8186681270599365s
Received healthy response to inference request in 2.5865631103515625s
Received healthy response to inference request in 2.809586524963379s
Received healthy response to inference request in 2.757011890411377s
Received healthy response to inference request in 2.6198887825012207s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.672651767730713s
Received healthy response to inference request in 2.690838098526001s
Received healthy response to inference request in 9.222352266311646s
Received healthy response to inference request in 2.584473133087158s
Received healthy response to inference request in 3.1051084995269775s
Received healthy response to inference request in 2.6060709953308105s
2026-03-27T23:52:28.430949+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
2026-03-27T23:52:32.927573+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 15.666301012039185s
Received healthy response to inference request in 2.6122050285339355s
Received healthy response to inference request in 2.7384588718414307s
Received healthy response to inference request in 2.6058297157287598s
Received healthy response to inference request in 2.5701797008514404s
Received healthy response to inference request in 2.6800100803375244s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.7650413513183594s
Received healthy response to inference request in 2.690929412841797s
Received healthy response to inference request in 2.6345834732055664s
Received healthy response to inference request in 2.7308053970336914s
30 requests
7 failed requests
5th percentile: 2.5766117453575133
10th percentile: 2.586354112625122
20th percentile: 2.6109782218933106
30th percentile: 2.666382098197937
40th percentile: 2.7495906829833983
50th percentile: 2.8141273260116577
60th percentile: 3.189289808273315
70th percentile: 9.442782616615295
80th percentile: 20.239820671081542
90th percentile: 20.273963618278504
95th percentile: 20.32098380327225
99th percentile: 20.321921298503877
mean time: 7.922867290178934
%s, retrying in %s seconds...
Received healthy response to inference request in 2.9230899810791016s
Received healthy response to inference request in 2.6141116619110107s
Received healthy response to inference request in 2.46197509765625s
Received healthy response to inference request in 2.634211778640747s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.599557399749756s
Received healthy response to inference request in 2.7064599990844727s
Received healthy response to inference request in 2.5394973754882812s
Received healthy response to inference request in 2.7247793674468994s
Received healthy response to inference request in 2.507835865020752s
2026-03-27T23:53:28.599963+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
Received healthy response to inference request in 2.5129177570343018s
Received healthy response to inference request in 2.4959890842437744s
Received healthy response to inference request in 2.566108226776123s
2026-03-27T23:53:33.091624+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
Received healthy response to inference request in 2.9232845306396484s
Received healthy response to inference request in 2.5926058292388916s
Received healthy response to inference request in 2.6498048305511475s
Received healthy response to inference request in 2.7421395778656006s
Received healthy response to inference request in 2.660457134246826s
Received healthy response to inference request in 2.881242513656616s
Received healthy response to inference request in 2.586869478225708s
Received healthy response to inference request in 2.669576644897461s
Received healthy response to inference request in 2.5689709186553955s
Received healthy response to inference request in 2.6118721961975098s
Received healthy response to inference request in 2.6603710651397705s
Received healthy response to inference request in 2.815077304840088s
Received healthy response to inference request in 2.5684309005737305s
Received healthy response to inference request in 2.8343935012817383s
Received healthy response to inference request in 2.7394895553588867s
Received healthy response to inference request in 2.5640223026275635s
Received healthy response to inference request in 2.7575244903564453s
30 requests
10 failed requests
5th percentile: 2.5387616753578186
10th percentile: 2.5899560689926147
20th percentile: 2.6720367431640626
30th percentile: 2.70180082321167
40th percentile: 2.7513705253601075
50th percentile: 2.824735403060913
60th percentile: 12.38780255317687
70th percentile: 20.259937119483947
80th percentile: 20.264689874649047
90th percentile: 20.286847233772278
95th percentile: 20.305948877334593
99th percentile: 20.361576795578003
mean time: 9.767171939214071
%s, retrying in %s seconds...
Received healthy response to inference request in 2.594536066055298s
Received healthy response to inference request in 2.5024020671844482s
Received healthy response to inference request in 2.57817006111145s
Received healthy response to inference request in 2.654994010925293s
Received healthy response to inference request in 9.630101919174194s
Received healthy response to inference request in 2.5901505947113037s
Received healthy response to inference request in 2.8481600284576416s
Received healthy response to inference request in 2.743805170059204s
Received healthy response to inference request in 2.5824737548828125s
Received healthy response to inference request in 2.686833381652832s
Received healthy response to inference request in 2.537348985671997s
Received healthy response to inference request in 2.60481333732605s
Received healthy response to inference request in 2.6377761363983154s
Received healthy response to inference request in 2.6352908611297607s
Received healthy response to inference request in 2.540968656539917s
Received healthy response to inference request in 2.7183890342712402s
Received healthy response to inference request in 2.5582592487335205s
Received healthy response to inference request in 2.7095532417297363s
Received healthy response to inference request in 2.5438156127929688s
2026-03-27T23:54:28.778676+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_49011_v2
Received healthy response to inference request in 2.6584248542785645s
Received healthy response to inference request in 2.514512300491333s
Received healthy response to inference request in 2.631533622741699s
30 requests
0 failed requests
5th percentile: 2.4988749265670775
10th percentile: 2.5072924852371217
Received healthy response to inference request in 2.6879777908325195s
20th percentile: 2.5675491809844972
30th percentile: 2.5842596530914306
40th percentile: 2.5975488662719726
50th percentile: 2.622822642326355
60th percentile: 2.6410964488983155
2026-03-27T23:54:33.348592+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v1
70th percentile: 2.6590087175369264
80th percentile: 2.691377353668213
90th percentile: 2.7399211168289184
95th percentile: 2.8424118161201473
99th percentile: 2.9232281112670897
mean time: 2.633562175432841
Pipeline stage StressChecker completed in 328.84s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.603571653366089s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.13s
Shutdown handler de-registered
chaiml-qwen-bobo-dpo-ju_49011_v2 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 2.530895709991455s
Received healthy response to inference request in 2.766774892807007s
Received healthy response to inference request in 2.575197219848633s
Received healthy response to inference request in 2.5877292156219482s
Received healthy response to inference request in 2.9619216918945312s
Received healthy response to inference request in 2.5580849647521973s
Received healthy response to inference request in 2.606135368347168s
Received healthy response to inference request in 2.759220600128174s
Received healthy response to inference request in 2.61621356010437s
Received healthy response to inference request in 3.7900514602661133s
Received healthy response to inference request in 2.769911050796509s
Received healthy response to inference request in 2.8557076454162598s
Received healthy response to inference request in 2.780940532684326s
Received healthy response to inference request in 2.6956965923309326s
Received healthy response to inference request in 2.5861763954162598s
Received healthy response to inference request in 2.5895376205444336s
Received healthy response to inference request in 2.640425443649292s
Received healthy response to inference request in 2.619459867477417s
Received healthy response to inference request in 2.6438398361206055s
30 requests
0 failed requests
5th percentile: 2.533799684047699
10th percentile: 2.540606689453125
20th percentile: 2.558224391937256
30th percentile: 2.5850656032562256
40th percentile: 2.597958040237427
50th percentile: 2.6178367137908936
60th percentile: 2.6417912006378175
70th percentile: 2.714753794670105
80th percentile: 2.772116947174072
90th percentile: 2.866329050064087
95th percentile: 3.4173930644988992
99th percentile: 7.9364872860908555
mean time: 2.9206295251846313
Pipeline stage StressChecker completed in 390.34s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.72s
Shutdown handler de-registered
chaiml-qwen-bobo-dpo-ju_56781_v1 status is now deployed due to DeploymentManager action
chaiml-qwen-bobo-dpo-ju_56781_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen-bobo-dpo-ju_56781_v1 status is now torndown due to DeploymentManager action