developer_uid: chai_backend_admin
submission_id: chaiml-merged-qwen-35-_39140_v16
model_name: chaiml-merged-qwen-35-_39140_v16
model_group: ChaiML/merged_qwen_35_dp
status: torndown
timestamp: 2026-03-31T14:41:49+00:00
num_battles: 10165
num_wins: 5528
celo_rating: 1324.94
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/merged_qwen_35_dpo_lower_lr_v
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-merged-qwen-35-_39140_v16
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/merged_qwen_35_dpo_lower_lr_v
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5438268568617807
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['###', '<|im_start|>', '<|im_end|>', '</s>', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-merged-qwen-35-39140-v16-uploader
Waiting for job on chaiml-merged-qwen-35-39140-v16-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-dpo-ju-56781-v3-uploader
Waiting for job on chaiml-qwen-bobo-dpo-ju-56781-v3-uploader to finish
chaiml-merged-qwen-35-39140-v16-uploader: Using quantization_mode: none
chaiml-merged-qwen-35-39140-v16-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v...
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: Using quantization_mode: none
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: Downloading snapshot of ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged...
chaiml-merged-qwen-35-39140-v16-uploader: Downloaded in 32.059s
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: Downloaded in 29.691s
2026-03-28T04:53:05.423622+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
2026-03-28T04:53:05.908252+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
chaiml-merged-qwen-35-39140-v16-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 53.191s
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: Processed model ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged in 49.981s
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/.gitattributes
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/model.safetensors.index.json
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/chat_template.jinja
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/generation_config.json
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/config.json
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/README.md
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/tokenizer_config.json
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/tokenizer.json
chaiml-merged-qwen-35-39140-v16-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-39140-v16-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v16-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-39140-v16-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-39140-v16-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-39140-v16-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v16-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v16-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v16-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v16-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v16-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v16-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v16-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v16-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-39140-v16-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-39140-v16-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-39140-v16-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-39140-v16-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-merged-qwen-35-39140-v16-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/.gitattributes
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/generation_config.json
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/tokenizer_config.json
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/config.json
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/model.safetensors.index.json
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/README.md
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/tokenizer.json
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/model-00002-of-00002.safetensors
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/model-00002-of-00002.safetensors
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-merged-qwen-35-39140-v17-uploader
Waiting for job on chaiml-merged-qwen-35-39140-v17-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-bobo-dpo-ju-56781-v4-uploader
Waiting for job on chaiml-qwen-bobo-dpo-ju-56781-v4-uploader to finish
2026-03-28T04:54:05.574884+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
2026-03-28T04:54:06.037731+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
chaiml-merged-qwen-35-39140-v17-uploader: Using quantization_mode: none
chaiml-merged-qwen-35-39140-v17-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v...
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: Using quantization_mode: none
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: Downloading snapshot of ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged...
chaiml-qwen-bobo-dpo-ju-56781-v3-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v3/default/model-00001-of-00002.safetensors
chaiml-merged-qwen-35-39140-v16-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v16/default/model-00001-of-00002.safetensors
Job chaiml-merged-qwen-35-39140-v16-uploader completed after 144.45s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-39140-v16-uploader
Pipeline stage VLLMUploader completed in 145.35s
Job chaiml-qwen-bobo-dpo-ju-56781-v3-uploader completed after 144.47s with status: succeeded
run pipeline stage %s
Stopping job with name chaiml-qwen-bobo-dpo-ju-56781-v3-uploader
Running pipeline stage VLLMTemplater
Pipeline stage VLLMUploader completed in 145.44s
run pipeline stage %s
Pipeline stage VLLMTemplater completed in 0.48s
Running pipeline stage VLLMTemplater
run pipeline stage %s
Running pipeline stage VLLMDeployer
Pipeline stage VLLMTemplater completed in 0.43s
run pipeline stage %s
Creating inference service chaiml-merged-qwen-35-39140-v16
Running pipeline stage VLLMDeployer
Waiting for inference service chaiml-merged-qwen-35-39140-v16 to be ready
Creating inference service chaiml-qwen-bobo-dpo-ju-56781-v3
Waiting for inference service chaiml-qwen-bobo-dpo-ju-56781-v3 to be ready
chaiml-merged-qwen-35-39140-v17-uploader: Downloaded in 31.379s
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: Downloaded in 30.347s
2026-03-28T04:54:57.250533+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
chaiml-merged-qwen-35-39140-v17-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 51.600s
chaiml-merged-qwen-35-39140-v17-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-39140-v17-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v17-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-39140-v17-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-39140-v17-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-39140-v17-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v17-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
2026-03-28T04:55:00.878369+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
chaiml-merged-qwen-35-39140-v17-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v17-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v17-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v17-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v17-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v17-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-39140-v17-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-39140-v17-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-39140-v17-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-39140-v17-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-39140-v17-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: Processed model ChaiML/qwen_bobo_dpo_judge_plus_reward_700_merged in 50.381s
chaiml-merged-qwen-35-39140-v17-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/config.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/README.md
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/model.safetensors.index.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/tokenizer_config.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/generation_config.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/.gitattributes
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/tokenizer.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
2026-03-28T04:55:05.801084+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
2026-03-28T04:55:06.259642+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/chat_template.jinja
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/README.md
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/config.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/tokenizer_config.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/.gitattributes
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/model.safetensors.index.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/generation_config.json
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/tokenizer.json
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/model-00002-of-00002.safetensors
2026-03-28T04:55:57.504039+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
2026-03-28T04:56:01.196839+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
2026-03-28T04:56:06.088382+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
2026-03-28T04:56:06.565732+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
chaiml-merged-qwen-35-39140-v17-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-39140-v17/default/model-00001-of-00002.safetensors
chaiml-qwen-bobo-dpo-ju-56781-v4-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-qwen-bobo-dpo-ju-56781-v4/default/model-00001-of-00002.safetensors
Job chaiml-merged-qwen-35-39140-v17-uploader completed after 148.24s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-39140-v17-uploader
Pipeline stage VLLMUploader completed in 149.41s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.48s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-merged-qwen-35-39140-v17
Waiting for inference service chaiml-merged-qwen-35-39140-v17 to be ready
Job chaiml-qwen-bobo-dpo-ju-56781-v4-uploader completed after 148.76s with status: succeeded
Stopping job with name chaiml-qwen-bobo-dpo-ju-56781-v4-uploader
Pipeline stage VLLMUploader completed in 150.06s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.46s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-qwen-bobo-dpo-ju-56781-v4
Waiting for inference service chaiml-qwen-bobo-dpo-ju-56781-v4 to be ready
2026-03-28T04:56:57.843753+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
2026-03-28T04:57:01.479100+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
2026-03-28T04:57:06.449485+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
2026-03-28T04:57:06.825691+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
Inference service chaiml-merged-qwen-35-39140-v16 ready after 180.3806266784668s
Pipeline stage VLLMDeployer completed in 181.62s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service chaiml-qwen-bobo-dpo-ju-56781-v3 ready after 190.49751830101013s
Pipeline stage VLLMDeployer completed in 191.64s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T04:57:58.141946+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
2026-03-28T04:58:02.108055+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T04:58:06.756020+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
2026-03-28T04:58:07.189632+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T04:58:58.436904+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
2026-03-28T04:59:02.485204+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
2026-03-28T04:59:07.101025+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
Received unhealthy response to inference request!
2026-03-28T04:59:07.498259+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
Received healthy response to inference request in 10.33986783027649s
Received healthy response to inference request in 10.460179328918457s
Inference service chaiml-merged-qwen-35-39140-v17 ready after 170.33673691749573s
Pipeline stage VLLMDeployer completed in 171.65s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.6748673915863037s
Received healthy response to inference request in 18.094779014587402s
Inference service chaiml-qwen-bobo-dpo-ju-56781-v4 ready after 170.35048580169678s
Pipeline stage VLLMDeployer completed in 173.48s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 11.984206199645996s
Received healthy response to inference request in 3.483868360519409s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.7046947479248047s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.649782419204712s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.7782952785491943s
Received healthy response to inference request in 3.974207639694214s
Received healthy response to inference request in 3.7482171058654785s
2026-03-28T04:59:58.741037+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T05:00:02.786533+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T05:00:07.486396+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
2026-03-28T05:00:07.813701+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
Received unhealthy response to inference request!
Received healthy response to inference request in 10.556436777114868s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.094933748245239s
Received healthy response to inference request in 4.04064154624939s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 10.344270467758179s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.697920560836792s
Received healthy response to inference request in 3.7036123275756836s
2026-03-28T05:00:59.132868+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
Received healthy response to inference request in 3.8741281032562256s
2026-03-28T05:01:03.136584+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.862450122833252s
2026-03-28T05:01:08.364387+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
2026-03-28T05:01:08.629564+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.375202655792236s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.7002782821655273s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 10.11241340637207s
Received healthy response to inference request in 3.6962032318115234s
Received healthy response to inference request in 3.721998453140259s
Received healthy response to inference request in 10.277453422546387s
Received healthy response to inference request in 3.5765812397003174s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received healthy response to inference request in 3.7413158416748047s
Received unhealthy response to inference request!
Received healthy response to inference request in 3.729698419570923s
Received healthy response to inference request in 3.6619338989257812s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.998661518096924s
Received healthy response to inference request in 10.374496221542358s
Received healthy response to inference request in 4.353023052215576s
2026-03-28T05:01:59.491829+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
Received healthy response to inference request in 16.60824751853943s
2026-03-28T05:02:03.549278+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
Received healthy response to inference request in 10.08714771270752s
Received healthy response to inference request in 16.190463304519653s
30 requests
7 failed requests
5th percentile: 3.609521770477295
2026-03-28T05:02:08.700890+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
10th percentile: 3.6723588943481444
2026-03-28T05:02:09.084260+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
20th percentile: 3.69980673789978
30th percentile: 3.7168073415756226
40th percentile: 3.766264009475708
50th percentile: 3.9241678714752197
Received healthy response to inference request in 12.572139978408813s
60th percentile: 6.491819334030143
70th percentile: 10.400201153755187
Received healthy response to inference request in 9.145804643630981s
80th percentile: 20.534431982040406
90th percentile: 20.592734622955323
95th percentile: 20.613286983966827
99th percentile: 20.89108971595764
mean time: 8.8464812596639
%s, retrying in %s seconds...
Received healthy response to inference request in 2.77660870552063s
Received healthy response to inference request in 3.758755683898926s
Received healthy response to inference request in 3.621001720428467s
Received healthy response to inference request in 3.747352361679077s
Received healthy response to inference request in 3.5661025047302246s
Received healthy response to inference request in 3.6487982273101807s
Received healthy response to inference request in 3.817072629928589s
Received healthy response to inference request in 3.708589792251587s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received healthy response to inference request in 3.4987874031066895s
Received unhealthy response to inference request!
Received healthy response to inference request in 3.702352285385132s
Received healthy response to inference request in 3.757333278656006s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received healthy response to inference request in 3.7840933799743652s
Received unhealthy response to inference request!
Received healthy response to inference request in 3.6723573207855225s
Received healthy response to inference request in 2.523175001144409s
Received healthy response to inference request in 3.570496082305908s
Received healthy response to inference request in 3.7139155864715576s
Received healthy response to inference request in 3.910162925720215s
Received healthy response to inference request in 3.613947629928589s
Received healthy response to inference request in 3.709713935852051s
Received healthy response to inference request in 3.4676003456115723s
Received healthy response to inference request in 3.7030081748962402s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.922036647796631s
Received healthy response to inference request in 9.494653463363647s
Received healthy response to inference request in 3.74238657951355s
Received healthy response to inference request in 3.59603214263916s
Received healthy response to inference request in 2.6519153118133545s
Received healthy response to inference request in 3.6417176723480225s
Received healthy response to inference request in 2.565387010574341s
Received healthy response to inference request in 3.6780147552490234s
30 requests
9 failed requests
Received healthy response to inference request in 9.270568370819092s
5th percentile: 3.5819873094558714
10th percentile: 3.612156081199646
2026-03-28T05:02:59.819715+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
20th percentile: 3.674798583984375
Received healthy response to inference request in 3.4863054752349854s
30th percentile: 3.7314980268478393
Received healthy response to inference request in 2.673672914505005s
40th percentile: 3.7739583015441895
50th percentile: 4.17584228515625
60th percentile: 10.429136991500854
70th percentile: 18.82703244686126
80th percentile: 20.53815336227417
Received healthy response to inference request in 2.598759889602661s
90th percentile: 20.57506721019745
95th percentile: 20.60013689994812
99th percentile: 20.63347778081894
mean time: 10.23081289132436
Received healthy response to inference request in 2.628445863723755s
2026-03-28T05:03:03.871526+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
%s, retrying in %s seconds...
Received healthy response to inference request in 3.7442259788513184s
Received healthy response to inference request in 2.5860719680786133s
Received healthy response to inference request in 3.6666698455810547s
Received healthy response to inference request in 3.6001322269439697s
2026-03-28T05:03:09.227449+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
2026-03-28T05:03:09.594742+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
Received healthy response to inference request in 2.5698044300079346s
Received healthy response to inference request in 3.679028034210205s
Received healthy response to inference request in 3.7385025024414062s
Received healthy response to inference request in 3.0107016563415527s
Received healthy response to inference request in 2.6632633209228516s
Received healthy response to inference request in 3.71602463722229s
Received healthy response to inference request in 4.141466379165649s
Received healthy response to inference request in 2.536336660385132s
Received healthy response to inference request in 3.6840972900390625s
Received healthy response to inference request in 3.69761061668396s
Received healthy response to inference request in 2.601067066192627s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.7068593502044678s
Received healthy response to inference request in 2.6496036052703857s
Received healthy response to inference request in 3.8119356632232666s
Received healthy response to inference request in 2.5673718452453613s
Received healthy response to inference request in 2.799799680709839s
Received healthy response to inference request in 3.7979307174682617s
Received healthy response to inference request in 2.8270137310028076s
Received healthy response to inference request in 3.7149105072021484s
Received healthy response to inference request in 2.6068880558013916s
Received healthy response to inference request in 2.5798959732055664s
Received healthy response to inference request in 3.6461341381073s
Received healthy response to inference request in 3.6457183361053467s
Received healthy response to inference request in 2.737316131591797s
Received healthy response to inference request in 2.7374796867370605s
Received healthy response to inference request in 3.8743176460266113s
Received healthy response to inference request in 2.6019890308380127s
Received healthy response to inference request in 3.786052703857422s
Received healthy response to inference request in 2.6374669075012207s
Received healthy response to inference request in 2.644289016723633s
30 requests
8 failed requests
5th percentile: 2.5494093179702757
Received healthy response to inference request in 4.1925623416900635s
Received healthy response to inference request in 2.577812910079956s
10th percentile: 2.569362688064575
Received healthy response to inference request in 3.8320021629333496s
20th percentile: 2.6018046379089355
30th percentile: 2.6395360708236693
40th percentile: 2.6587241172790526
50th percentile: 2.7569624185562134
60th percentile: 3.370486164093016
70th percentile: 11.628731679916362
80th percentile: 20.548128032684325
90th percentile: 20.57436845302582
Received healthy response to inference request in 3.0245957374572754s
95th percentile: 20.640723073482512
Received healthy response to inference request in 3.8093888759613037s
99th percentile: 20.720896248817443
Received healthy response to inference request in 3.693725109100342s
mean time: 8.39210721651713
%s, retrying in %s seconds...
Received healthy response to inference request in 2.5992119312286377s
Received healthy response to inference request in 3.538461446762085s
Received healthy response to inference request in 2.4496428966522217s
Received healthy response to inference request in 3.7399659156799316s
Received healthy response to inference request in 2.787086248397827s
Received healthy response to inference request in 2.499699831008911s
Received healthy response to inference request in 3.774960517883301s
Received healthy response to inference request in 3.5503602027893066s
Received healthy response to inference request in 2.6566784381866455s
Received healthy response to inference request in 2.4815096855163574s
Received healthy response to inference request in 2.617281913757324s
Received healthy response to inference request in 3.749490976333618s
Received healthy response to inference request in 3.669093370437622s
Received healthy response to inference request in 2.511054039001465s
2026-03-28T05:04:00.419039+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v17
Received healthy response to inference request in 3.738640546798706s
Received healthy response to inference request in 3.226264715194702s
Received healthy response to inference request in 3.655841588973999s
2026-03-28T05:04:04.402913+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v4
Received healthy response to inference request in 2.5772461891174316s
Received healthy response to inference request in 3.7313268184661865s
Received healthy response to inference request in 3.7559995651245117s
Received healthy response to inference request in 9.353704452514648s
Received healthy response to inference request in 2.5778191089630127s
2026-03-28T05:04:09.628933+00:00 monitor updated for chaiml-merged-qwen-35-_39140_v16
Received healthy response to inference request in 2.6619038581848145s
2026-03-28T05:04:09.958479+00:00 monitor updated for chaiml-qwen-bobo-dpo-ju_56781_v3
Received healthy response to inference request in 3.8127951622009277s
Received healthy response to inference request in 3.796048164367676s
Received healthy response to inference request in 2.645338296890259s
Received healthy response to inference request in 2.929084062576294s
Received healthy response to inference request in 2.532857656478882s
Received healthy response to inference request in 3.844804048538208s
Received healthy response to inference request in 3.6888010501861572s
30 requests
0 failed requests
5th percentile: 3.5219951629638673
10th percentile: 3.5645282745361326
Received healthy response to inference request in 2.7404239177703857s
Received healthy response to inference request in 2.4996190071105957s
20th percentile: 3.637574481964111
30th percentile: 3.6651178359985352
40th percentile: 3.691755485534668
50th percentile: 3.7063610553741455
60th percentile: 3.7243473052978517
70th percentile: 3.7429383993148804
Received healthy response to inference request in 3.969595193862915s
80th percentile: 3.763077163696289
90th percentile: 3.8124493598937987
Received healthy response to inference request in 2.7373809814453125s
Received healthy response to inference request in 2.7060739994049072s
95th percentile: 3.825283873081207
30 requests
99th percentile: 4.051721756458282
9 failed requests
mean time: 3.706403144200643
5th percentile: 2.578750288486481
Pipeline stage StressChecker completed in 407.42s
10th percentile: 2.5968734979629517
run pipeline stage %s
20th percentile: 2.6334299087524413
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
30th percentile: 2.714737844467163
Received healthy response to inference request in 3.712684392929077s
Received healthy response to inference request in 2.6958675384521484s
run_pipeline:run_in_cloud %s
40th percentile: 2.7684213161468505
starting trigger_guanaco_pipeline args=%s
50th percentile: 2.9768399000167847
60th percentile: 9.362303733825684
triggered trigger_guanaco_pipeline args=%s
70th percentile: 17.500307059288012
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.39s
80th percentile: 20.59280652999878
Shutdown handler de-registered
90th percentile: 20.72016670703888
90th percentile: 20.72016670703888
chaiml-merged-qwen-35-_39140_v16 status is now deployed due to DeploymentManager action
chaiml-merged-qwen-35-_39140_v16 status is now inactive due to auto deactivation removed underperforming models
chaiml-merged-qwen-35-_39140_v16 status is now torndown due to DeploymentManager action