chaiml-qwen-bobo-dpo-ju_56781

developer_uid: rirv938
submission_id: chaiml-qwen-bobo-dpo-ju_56781_v5
model_name: chaiml-qwen-bobo-dpo-ju_56781_v5
model_group: ChaiML/qwen_bobo_dpo_jud
status: torndown
timestamp: 2026-03-31T20:51:55+00:00
num_battles: 10511
num_wins: 5411
celo_rating: 1302.37
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 1
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen-bobo-dpo-ju_56781_v5
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5147940253068214
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '###', '</s>', '<|im_end|>', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 1, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-dpo-ju-56781-v5-uploader
Waiting for job on chaiml-qwen-bobo-dpo-ju-56781-v5-uploader to finish
chaiml-merged-qwen-35-39140-v19-uploader: Using quantization_mode: none
chaiml-merged-qwen-35-39140-v19-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v...
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: Using quantization_mode: none
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: Downloading snapshot of ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged...
chaiml-merged-qwen-35-39140-v19-uploader: Downloaded in 30.494s
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: Downloaded in 30.388s
2026-03-28T16:37:51.670927+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
chaiml-merged-qwen-35-39140-v19-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 50.722s
chaiml-merged-qwen-35-39140-v19-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-39140-v19-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v19-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-39140-v19-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-39140-v19-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-39140-v19-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v19-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v19-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v19-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v19-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v19-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v19-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v19-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v19-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-39140-v19-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-39140-v19-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-39140-v19-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-39140-v19-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-merged-qwen-35-39140-v19-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/.gitattributes
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/config.json
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/generation_config.json
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/tokenizer_config.json
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/README.md
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/model.safetensors.index.json
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/tokenizer.json
2026-03-28T16:37:55.649797+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: Processed model ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged in 50.789s
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/.gitattributes
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/generation_config.json
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/chat_template.jinja
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/README.md
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/config.json
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/tokenizer_config.json
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/model.safetensors.index.json
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/tokenizer.json
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/model-00002-of-00002.safetensors
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/model-00002-of-00002.safetensors
2026-03-28T16:38:51.801542+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
2026-03-28T16:38:55.786005+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
2026-03-28T16:39:51.951730+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
2026-03-28T16:39:56.160266+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
chaiml-qwen-bobo-dpo-ju-56781-v5-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v5/default/model-00001-of-00002.safetensors
Job chaiml-qwen-bobo-dpo-ju-56781-v5-uploader completed after 195.05s with status: succeeded
Stopping job with name chaiml-qwen-bobo-dpo-ju-56781-v5-uploader
Pipeline stage VLLMUploader completed in 195.75s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.05s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-dpo-ju-56781-v5
Waiting for inference service chaiml-qwen-bobo-dpo-ju-56781-v5 to be ready
chaiml-merged-qwen-35-39140-v19-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v19/default/model-00001-of-00002.safetensors
Job chaiml-merged-qwen-35-39140-v19-uploader completed after 234.82s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-39140-v19-uploader
Pipeline stage VLLMUploader completed in 235.46s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.51s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-merged-qwen-35-39140-v19
Waiting for inference service chaiml-merged-qwen-35-39140-v19 to be ready
2026-03-28T16:40:52.097185+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
2026-03-28T16:40:56.310722+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
2026-03-28T16:41:52.274158+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
2026-03-28T16:41:56.458211+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
2026-03-28T16:42:52.430470+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
Inference service chaiml-qwen-bobo-dpo-ju-56781-v5 ready after 160.41443991661072s
Pipeline stage VLLMDeployer completed in 161.22s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-28T16:42:56.605455+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Inference service chaiml-merged-qwen-35-39140-v19 ready after 170.40716171264648s
Pipeline stage VLLMDeployer completed in 171.15s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-28T16:43:52.622382+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
2026-03-28T16:43:56.841070+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.925110340118408s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.7684004306793213s
Received healthy response to inference request in 9.310362815856934s
Received healthy response to inference request in 1.7825396060943604s
2026-03-28T16:44:52.790719+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
2026-03-28T16:44:57.003645+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.6799535751342773s
Received healthy response to inference request in 1.930558204650879s
Received healthy response to inference request in 1.7148959636688232s
Received healthy response to inference request in 1.9133384227752686s
Received healthy response to inference request in 1.8290619850158691s
Received healthy response to inference request in 1.7666699886322021s
Received healthy response to inference request in 1.9100539684295654s
Received healthy response to inference request in 1.7751493453979492s
Received healthy response to inference request in 1.7964723110198975s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.6303813457489014s
Received healthy response to inference request in 1.956568956375122s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T16:45:52.955235+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.8141396045684814s
2026-03-28T16:45:57.161839+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
Received healthy response to inference request in 1.6545052528381348s
Received healthy response to inference request in 12.674556016921997s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.9575293064117432s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received healthy response to inference request in 1.8108582496643066s
Received unhealthy response to inference request!
Received healthy response to inference request in 1.7850620746612549s
Received healthy response to inference request in 1.8231394290924072s
Received healthy response to inference request in 1.5194437503814697s
Received healthy response to inference request in 2.133146047592163s
Received healthy response to inference request in 1.7401163578033447s
Received healthy response to inference request in 1.6704249382019043s
Received healthy response to inference request in 1.7173049449920654s
Received healthy response to inference request in 1.6584270000457764s
Received healthy response to inference request in 1.7512052059173584s
Received healthy response to inference request in 1.7553331851959229s
Received healthy response to inference request in 1.7084243297576904s
Received healthy response to inference request in 1.7197866439819336s
Received healthy response to inference request in 1.7506258487701416s
Received healthy response to inference request in 1.6399035453796387s
Received healthy response to inference request in 2.4658141136169434s
30 requests
8 failed requests
5th percentile: 1.6579260587692262
10th percentile: 1.7055772542953491
20th percentile: 1.735554075241089
30th percentile: 1.762030553817749
40th percentile: 1.779583501815796
50th percentile: 1.8261007070541382
60th percentile: 1.9202263355255127
70th percentile: 10.749944043159477
80th percentile: 20.244726800918578
90th percentile: 20.266355323791505
95th percentile: 20.30113297700882
99th percentile: 20.34733698606491
mean time: 7.359298332532247
%s, retrying in %s seconds...
Received healthy response to inference request in 1.7958240509033203s
Received healthy response to inference request in 11.43604588508606s
Received healthy response to inference request in 1.7954623699188232s
Received healthy response to inference request in 1.777803897857666s
Received healthy response to inference request in 1.6450939178466797s
Received healthy response to inference request in 1.6715712547302246s
Received healthy response to inference request in 1.667684555053711s
Received healthy response to inference request in 1.652235507965088s
Received healthy response to inference request in 1.814000129699707s
Received healthy response to inference request in 1.5021729469299316s
2026-03-28T16:46:53.122261+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v19
Received healthy response to inference request in 1.7724602222442627s
Received healthy response to inference request in 1.808619499206543s
Received healthy response to inference request in 1.8015179634094238s
Received healthy response to inference request in 1.7700042724609375s
2026-03-28T16:46:57.317688+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v5
Received healthy response to inference request in 1.686396837234497s
Received healthy response to inference request in 1.7608249187469482s
Received healthy response to inference request in 1.7978236675262451s
Received healthy response to inference request in 1.716627597808838s
Received healthy response to inference request in 1.7927255630493164s
Received healthy response to inference request in 1.6791400909423828s
30 requests
7 failed requests
5th percentile: 1.6402157187461852
10th percentile: 1.65427827835083
20th percentile: 1.6713419914245606
30th percentile: 1.7188389301300049
Received healthy response to inference request in 1.672454833984375s
40th percentile: 1.7663325309753417
50th percentile: 1.796840786933899
60th percentile: 1.8711113452911374
70th percentile: 4.286311078071574
80th percentile: 20.244891452789307
90th percentile: 20.260136675834655
95th percentile: 20.268650114536285
99th percentile: 20.362899210453033
mean time: 6.652630225817362
%s, retrying in %s seconds...
Received healthy response to inference request in 1.7911102771759033s
Received healthy response to inference request in 1.8518016338348389s
Received healthy response to inference request in 1.730090618133545s
Received healthy response to inference request in 1.9435791969299316s
Received healthy response to inference request in 1.6211888790130615s
Received healthy response to inference request in 1.5160045623779297s
Received healthy response to inference request in 1.6762380599975586s
Received healthy response to inference request in 1.5473744869232178s
Received healthy response to inference request in 2.117649793624878s
Received healthy response to inference request in 1.789132833480835s
Received healthy response to inference request in 1.7541968822479248s
Received healthy response to inference request in 1.6901795864105225s
Received healthy response to inference request in 1.8215811252593994s
Received healthy response to inference request in 1.6915104389190674s
Received healthy response to inference request in 1.6132545471191406s
Received healthy response to inference request in 1.7930948734283447s
Received healthy response to inference request in 1.75628662109375s
Received healthy response to inference request in 1.760866403579712s
Received healthy response to inference request in 1.6363015174865723s
Received healthy response to inference request in 1.806788444519043s
Received healthy response to inference request in 1.7546730041503906s
Received healthy response to inference request in 1.7967066764831543s
Received healthy response to inference request in 1.8310785293579102s
Received healthy response to inference request in 1.805997610092163s
Received healthy response to inference request in 1.763603925704956s
Received healthy response to inference request in 1.7780895233154297s
Received healthy response to inference request in 1.7862353324890137s
Received healthy response to inference request in 1.8219578266143799s
Received healthy response to inference request in 1.7600293159484863s
Received healthy response to inference request in 1.3713488578796387s
Received healthy response to inference request in 1.7117371559143066s
Received healthy response to inference request in 2.0587003231048584s
Received healthy response to inference request in 1.827528476715088s
Received healthy response to inference request in 1.838881492614746s
Received healthy response to inference request in 1.7074270248413086s
Received healthy response to inference request in 1.7826952934265137s
Received healthy response to inference request in 2.472111701965332s
30 requests
0 failed requests
5th percentile: 1.6279895663261414
10th percentile: 1.644214677810669
20th percentile: 1.675481414794922
30th percentile: 1.7104441165924071
40th percentile: 1.7544825553894043
50th percentile: 1.7618166208267212
60th percentile: 1.7881853103637695
70th percentile: 1.7955708742141723
Received healthy response to inference request in 2.6623923778533936s
80th percentile: 1.8040143966674804
90th percentile: 1.8278834819793701
95th percentile: 1.9886927247047417
99th percentile: 2.369317748546601
mean time: 1.7791255633036296
Pipeline stage StressChecker completed in 285.95s
run pipeline stage %s
Received healthy response to inference request in 1.6949994564056396s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.23s
Shutdown handler de-registered
chaiml-qwen-bobo-dpo-ju_56781_v5 status is now deployed due to DeploymentManager action
chaiml-qwen-bobo-dpo-ju_56781_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen-bobo-dpo-ju_56781_v5 status is now torndown due to DeploymentManager action