developer_uid: rirv938
submission_id: chaiml-q235b-judge-dpo-_28112_v1
model_name: chaiml-q235b-judge-dpo-_28112_v1
model_group: ChaiML/q235b_judge_dpo_l
status: torndown
timestamp: 2026-04-03T03:32:13+00:00
num_battles: 10374
num_wins: 5435
celo_rating: 1325.04
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/q235b_judge_dpo_lr2-step450-merged
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-q235b-judge-dpo-_28112_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/q235b_judge_dpo_lr2-step450-merged
model_size: 19B
ranking_group: single
us_pacific_date: 2026-03-30
win_ratio: 0.5239059186427607
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '####', '<|im_end|>', '<|assistant|>', '<|user|>', '</think>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-judge-dpo-28112-v1-uploader
Waiting for job on chaiml-q235b-judge-dpo-28112-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-judge-dpo-l-6582-v1-uploader
Waiting for job on chaiml-q235b-judge-dpo-l-6582-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v32-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v32-uploader to finish
chaiml-q235b-judge-dpo-28112-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-28112-v1-uploader: Checking if ChaiML/q235b_judge_dpo_lr2-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-28112-v1-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo_lr2-step450-merged...
2026-03-30T21:22:41.109761+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:22:45.750510+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:22:50.854568+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:23:41.465132+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:23:46.793110+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:23:51.779189+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:24:42.330679+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:24:47.063241+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:24:52.377022+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:25:42.732362+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:25:47.247267+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:25:52.817817+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
chaiml-q235b-judge-dpo-28112-v1-uploader: Downloaded in 216.047s
chaiml-q235b-judge-dpo-28112-v1-uploader: Applying quantization...
chaiml-q235b-judge-dpo-28112-v1-uploader: 2026-03-30 14:26:06 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Downloaded in 220.225s
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Applying quantization...
chaiml-q235b-judge-dpo-l-6582-v1-uploader: 2026-03-30 14:26:08 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-q235b-judge-dpo-28112-v1-uploader: 2026-03-30 14:26:25 INFO base.py L366: using torch.bfloat16 for quantization tuning
chaiml-q235b-judge-dpo-28112-v1-uploader: 2026-03-30 14:26:29 INFO base.py L1145: start to compute imatrix
2026-03-30T21:26:42.925974+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
chaiml-q235b-judge-dpo-l-6582-v1-uploader: 2026-03-30 14:26:47 INFO base.py L366: using torch.bfloat16 for quantization tuning
2026-03-30T21:26:47.449591+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:26:52.999599+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
chaiml-q235b-judge-dpo-l-6582-v1-uploader: 2026-03-30 14:26:51 INFO base.py L1145: start to compute imatrix
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/local/lib/python3.12/dist-packages/torch/backends/cuda/__init__.py:131: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
chaiml-q235b-judge-dpo-28112-v1-uploader:   return torch._C._get_cublas_allow_tf32()
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:32.029000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:32.029000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:32.029000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1339             # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:32.029000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:32.029000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-l-6582-v1-uploader: /usr/local/lib/python3.12/dist-packages/torch/backends/cuda/__init__.py:131: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
chaiml-q235b-judge-dpo-l-6582-v1-uploader:   return torch._C._get_cublas_allow_tf32()
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:34.951000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
2026-03-30T21:27:43.323837+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:34.951000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:34.951000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56  # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:34.951000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-28112-v1-uploader: W0330 14:27:34.951000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2026-03-30T21:27:48.398367+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:27:53.882669+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
chaiml-q235b-judge-dpo-28112-v1-uploader: 2026-03-30 14:27:52 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:27:57.188000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:27:57.188000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:27:57.188000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1342             # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:27:57.188000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:27:57.188000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:28:00.245000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:28:00.245000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:28:00.245000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56  # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:28:00.245000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-l-6582-v1-uploader: W0330 14:28:00.245000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-l-6582-v1-uploader: 2026-03-30 14:28:19 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0
2026-03-30T21:28:44.914518+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:28:50.311283+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:28:56.070661+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:29:45.091197+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:29:51.382358+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:29:56.565874+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:30:45.529528+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:30:52.071064+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:30:57.283046+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:31:45.832990+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:31:52.452967+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:31:57.744587+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:32:46.044886+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:32:52.838835+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:32:58.122423+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:33:46.452649+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:33:53.057919+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:33:58.315863+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:34:46.859941+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
chaiml-kimid-v9-opusdv-23365-v32-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v32-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v9-opusdv-23365-v32-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v32-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
2026-03-30T21:34:53.250846+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:34:58.509401+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:35:47.246564+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:35:53.996536+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:35:58.931850+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:36:48.417609+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:36:54.394313+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:36:59.345650+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
chaiml-kimid-v9-opusdv-23365-v32-uploader: Downloaded in 158.727s
chaiml-kimid-v9-opusdv-23365-v32-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 159.381s
chaiml-kimid-v9-opusdv-23365-v32-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-23365-v32-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v32-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v9-opusdv-23365-v32-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v9-opusdv-23365-v32-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v9-opusdv-23365-v32-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v32-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v32-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v32-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v32-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v32-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v32-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v32-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v32-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v9-opusdv-23365-v32-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v32-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v9-opusdv-23365-v32-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v9-opusdv-23365-v32-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-23365-v32-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/chat_template.jinja
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/generation_config.json
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/added_tokens.json
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/config.json
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/tokenizer_config.json
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/.gitattributes
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/quantization_config.json
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/special_tokens_map.json
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/merges.txt
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/vocab.json
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/tokenizer.json
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model.safetensors.index.json
2026-03-30T21:37:48.789137+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:37:54.785993+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00010-of-00027.safetensors
2026-03-30T21:37:59.788002+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v32-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v32/default/model-00024-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-23365-v32-uploader completed after 1000.22s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v32-uploader
Pipeline stage VLLMUploader completed in 1001.24s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.80s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v32
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v32 to be ready
2026-03-30T21:38:49.179596+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:38:55.852875+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:39:00.128694+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:39:49.630647+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:39:56.283715+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:40:00.603429+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:40:49.965458+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:40:56.674891+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:41:00.895064+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
2026-03-30T21:41:50.396067+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:41:56.911527+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:42:01.139881+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
Inference service chaiml-kimid-v9-opusdv-23365-v32 ready after 220.5783543586731s
Pipeline stage VLLMDeployer completed in 221.74s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.8219027519226074s
Received healthy response to inference request in 1.4421212673187256s
Received healthy response to inference request in 3.4451913833618164s
Received healthy response to inference request in 1.4222114086151123s
Received healthy response to inference request in 3.663100242614746s
Received healthy response to inference request in 1.433058500289917s
Received healthy response to inference request in 1.790069341659546s
Received healthy response to inference request in 1.613365888595581s
Received healthy response to inference request in 3.568031072616577s
Received healthy response to inference request in 1.5532805919647217s
Received healthy response to inference request in 3.6992568969726562s
2026-03-30T21:42:50.851762+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
Received healthy response to inference request in 1.5393035411834717s
Received healthy response to inference request in 1.5227515697479248s
Received healthy response to inference request in 1.446864366531372s
2026-03-30T21:42:57.145992+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
Received healthy response to inference request in 1.5415043830871582s
Received healthy response to inference request in 1.6500029563903809s
Received healthy response to inference request in 1.4369933605194092s
2026-03-30T21:43:01.375868+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v32
Received healthy response to inference request in 1.4787180423736572s
Received healthy response to inference request in 1.5857746601104736s
Received healthy response to inference request in 1.6352288722991943s
Received healthy response to inference request in 1.4481208324432373s
Received healthy response to inference request in 1.6389946937561035s
Received healthy response to inference request in 1.5774939060211182s
Received healthy response to inference request in 1.4591901302337646s
Received healthy response to inference request in 1.4556236267089844s
Received healthy response to inference request in 1.513584852218628s
Received healthy response to inference request in 1.5025439262390137s
Received healthy response to inference request in 1.7804951667785645s
Received healthy response to inference request in 1.4844324588775635s
Received healthy response to inference request in 1.769895315170288s
30 requests
0 failed requests
5th percentile: 1.4348291873931884
10th percentile: 1.441608476638794
20th percentile: 1.454123067855835
30th percentile: 1.4827181339263915
40th percentile: 1.519084882736206
50th percentile: 1.54739248752594
60th percentile: 1.5968111515045165
70th percentile: 1.6422971725463866
80th percentile: 1.7824100017547608
90th percentile: 3.577537989616394
95th percentile: 3.6829864025115966
99th percentile: 3.786335453987122
mean time: 1.897303533554077
Pipeline stage StressChecker completed in 70.98s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.51s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_23365_v32 status is now deployed due to DeploymentManager action
2026-03-30T21:43:51.090637+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:43:57.392218+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:44:51.401104+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:44:58.456013+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:45:52.341086+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:45:58.830409+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:46:52.749813+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:46:59.226980+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:47:54.402851+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:48:02.084865+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:48:54.588031+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:49:02.485052+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:49:54.774787+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:50:02.683497+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:50:54.959670+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:51:02.876195+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:51:55.158127+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:52:03.065558+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:52:55.352978+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:53:03.250666+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:53:55.745431+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:54:04.408101+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:54:56.287786+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:55:04.794094+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:55:56.702535+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:56:05.181111+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:56:57.130455+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:57:05.591191+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:57:58.000544+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:58:06.599133+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:58:58.394131+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T21:59:06.778343+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T21:59:58.769409+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:00:06.982777+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:00:59.150204+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:01:07.379712+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:01:59.632378+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:02:08.182208+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:03:00.114049+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:03:09.273099+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:04:00.541482+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:04:09.688728+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:05:01.138403+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:05:10.765017+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:06:01.686640+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:06:11.297724+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:07:01.876958+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:07:11.501263+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:08:03.016148+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:08:11.687008+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:09:03.207154+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:09:11.876352+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:10:03.631486+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:10:12.221260+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:11:04.050573+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:11:12.654997+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:12:05.209733+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:12:14.238990+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:13:07.950296+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:13:16.241111+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:14:08.144080+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:14:17.337129+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:15:08.335018+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:15:17.748321+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:16:08.643027+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:16:17.946625+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:17:08.980487+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:17:18.128050+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:18:10.441524+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:18:19.255411+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:19:10.722962+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:19:19.881085+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:20:11.448504+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:20:20.272624+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:21:12.479099+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:21:20.455364+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:22:12.911639+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:22:21.419339+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:23:14.156308+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:23:21.606921+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:24:14.568964+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:24:21.803860+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:25:15.011176+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:25:22.008658+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:26:15.454491+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:26:29.384403+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:27:15.927149+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:27:30.129000+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:28:16.755195+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:28:30.567165+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:29:17.325225+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:29:31.014866+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:30:17.673443+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:30:31.469089+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:31:17.902175+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:31:32.031129+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:32:18.086349+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:32:32.491984+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:33:18.545708+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:33:32.922973+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:34:18.749565+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:34:33.285364+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:35:19.156157+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:35:33.767612+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:36:19.568578+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:36:34.172034+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:37:20.636536+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:37:34.570351+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:38:21.377681+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:38:35.724670+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:39:22.550852+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:39:36.135376+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:40:23.005190+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:40:36.336119+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:41:30.737267+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:41:36.527344+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:42:31.168313+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:42:36.760404+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:43:31.595480+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:43:37.186630+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:44:31.987300+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:44:37.582355+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-q235b-judge-dpo-28112-v1-uploader: Checking if ChaiML/q235b_judge_dpo_lr2-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-28112-v1-uploader: Creating repo ChaiML/q235b_judge_dpo_lr2-step450-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------- 2026-03-30 15:44:31 (0:00:00) ----------
chaiml-q235b-judge-dpo-28112-v1-uploader: Files:   hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+27 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-28112-v1-uploader: Workers: hashing: 27 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 98
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-28112-v1-uploader: 







chaiml-q235b-judge-dpo-28112-v1-uploader: ---------- 2026-03-30 15:45:31 (0:01:00) ----------
chaiml-q235b-judge-dpo-28112-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 2/28 (1.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-28112-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------------------------------------------------
2026-03-30T22:45:33.028229+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:45:38.881968+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-q235b-judge-dpo-28112-v1-uploader:                              







2026-03-30T22:46:34.009769+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------- 2026-03-30 15:46:31 (0:02:00) ----------
chaiml-q235b-judge-dpo-28112-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-28112-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------------------------------------------------
2026-03-30T22:46:39.330984+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:47:35.734605+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
chaiml-q235b-judge-dpo-28112-v1-uploader:                              







2026-03-30T22:47:40.829855+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------- 2026-03-30 15:47:31 (0:03:00) ----------
chaiml-q235b-judge-dpo-28112-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 11/28 (46.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-28112-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 17 | committing: 0 | waiting: 109
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-28112-v1-uploader:                              







chaiml-q235b-judge-dpo-28112-v1-uploader: ---------- 2026-03-30 15:48:31 (0:04:00) ----------
chaiml-q235b-judge-dpo-28112-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-28112-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------------------------------------------------
2026-03-30T22:48:36.670621+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:48:42.000123+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-q235b-judge-dpo-28112-v1-uploader:                              







chaiml-q235b-judge-dpo-28112-v1-uploader: ---------- 2026-03-30 15:49:31 (0:05:00) ----------
chaiml-q235b-judge-dpo-28112-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-28112-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-q235b-judge-dpo-28112-v1-uploader: ---------------------------------------------------
2026-03-30T22:49:37.017615+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:49:42.210790+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-q235b-judge-dpo-28112-v1-uploader:                              Processed model ChaiML/q235b_judge_dpo_lr2-step450-merged in 5273.040s
chaiml-q235b-judge-dpo-28112-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-28112-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-judge-dpo-28112-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
2026-03-30T22:50:37.230680+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-28112-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-28112-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-28112-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-28112-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-judge-dpo-28112-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-judge-dpo-28112-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-judge-dpo-28112-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-judge-dpo-28112-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-judge-dpo-28112-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/chat_template.jinja
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/added_tokens.json
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/merges.txt
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/quantization_config.json
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/config.json
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/special_tokens_map.json
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/tokenizer_config.json
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/generation_config.json
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model.safetensors.index.json
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/vocab.json
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/tokenizer.json
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Checking if ChaiML/q235b_judge_dpo_lr2-step225-merged-W4A16 already exists in ChaiML
2026-03-30T22:50:42.419196+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Creating repo ChaiML/q235b_judge_dpo_lr2-step225-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------- 2026-03-30 15:50:34 (0:00:00) ----------
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Files:   hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+27 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Workers: hashing: 27 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 98
chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00001-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00018-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00022-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00023-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00024-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-28112-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-28112-v1/default/model-00012-of-00027.safetensors
Job chaiml-q235b-judge-dpo-28112-v1-uploader completed after 5380.73s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-28112-v1-uploader
Pipeline stage VLLMUploader completed in 5382.14s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.92s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-28112-v1
Waiting for inference service chaiml-q235b-judge-dpo-28112-v1 to be ready
2026-03-30T22:51:37.705853+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:51:42.652672+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-q235b-judge-dpo-l-6582-v1-uploader: 







chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------- 2026-03-30 15:51:34 (0:01:00) ----------
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 2/28 (1.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-l-6582-v1-uploader:                              







chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------- 2026-03-30 15:52:34 (0:02:00) ----------
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------------------------------------------------
2026-03-30T22:52:38.899579+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
2026-03-30T22:52:42.864969+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
chaiml-q235b-judge-dpo-l-6582-v1-uploader:                              







chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------- 2026-03-30 15:53:34 (0:03:00) ----------
2026-03-30T22:53:39.422448+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------------------------------------------------
2026-03-30T22:53:43.383211+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
2026-03-30T22:54:39.872481+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
chaiml-q235b-judge-dpo-l-6582-v1-uploader:                              







chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------- 2026-03-30 15:54:34 (0:04:00) ----------
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------------------------------------------------
2026-03-30T22:54:43.857949+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
Inference service chaiml-q235b-judge-dpo-28112-v1 ready after 220.5538387298584s
Pipeline stage VLLMDeployer completed in 222.94s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.16276741027832s
Received healthy response to inference request in 3.4611244201660156s
Received healthy response to inference request in 3.456700086593628s
Received healthy response to inference request in 3.775786876678467s
Received healthy response to inference request in 1.4851162433624268s
Received healthy response to inference request in 1.6003234386444092s
2026-03-30T22:55:40.172203+00:00 monitor updated for chaiml-q235b-judge-dpo-_28112_v1
Received healthy response to inference request in 3.628803014755249s
2026-03-30T22:55:44.545007+00:00 monitor updated for chaiml-q235b-judge-dpo-l_6582_v1
Received healthy response to inference request in 1.6537015438079834s
Received healthy response to inference request in 1.4528110027313232s
Received healthy response to inference request in 1.557919979095459s
Received healthy response to inference request in 1.5206005573272705s
Received healthy response to inference request in 1.4718847274780273s
Received healthy response to inference request in 1.764674186706543s
Received healthy response to inference request in 1.4802849292755127s
Received healthy response to inference request in 1.4339399337768555s
Received healthy response to inference request in 1.5184109210968018s
Received healthy response to inference request in 1.4568252563476562s
Received healthy response to inference request in 1.5037829875946045s
Received healthy response to inference request in 1.5246810913085938s
Received healthy response to inference request in 1.481490135192871s
Received healthy response to inference request in 1.620784044265747s
Received healthy response to inference request in 1.4789307117462158s
Received healthy response to inference request in 1.5318822860717773s
Received healthy response to inference request in 1.5478816032409668s
Received healthy response to inference request in 1.9290854930877686s
Received healthy response to inference request in 1.4826226234436035s
Received healthy response to inference request in 1.454885482788086s
Received healthy response to inference request in 1.4874467849731445s
Received healthy response to inference request in 1.8117039203643799s
Received healthy response to inference request in 1.7056884765625s
30 requests
0 failed requests
5th percentile: 1.4537445187568665
10th percentile: 1.4566312789916993
20th percentile: 1.4800140857696533
30th percentile: 1.4843681573867797
40th percentile: 1.512559747695923
50th percentile: 1.5282816886901855
60th percentile: 1.574881362915039
70th percentile: 1.6692976236343382
80th percentile: 1.835180234909058
90th percentile: 3.4778922796249394
95th percentile: 3.7096441388130184
99th percentile: 4.050543055534363
mean time: 1.9147513389587403
Pipeline stage StressChecker completed in 85.79s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
chaiml-q235b-judge-dpo-l-6582-v1-uploader:                              







run_pipeline:run_in_cloud %s
chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------- 2026-03-30 15:56:34 (0:06:00) ----------
starting trigger_guanaco_pipeline args=%s
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-l-6582-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-q235b-judge-dpo-l-6582-v1-uploader: ---------------------------------------------------
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.75s
Shutdown handler de-registered
chaiml-q235b-judge-dpo-_28112_v1 status is now deployed due to DeploymentManager action
chaiml-q235b-judge-dpo-_28112_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-q235b-judge-dpo-_28112_v1 status is now torndown due to DeploymentManager action