chaiml-q235b-judge-dpo-_47447

developer_uid: rirv938
submission_id: chaiml-q235b-judge-dpo-_47447_v1
model_name: chaiml-q235b-judge-dpo-_47447_v1
model_group: ChaiML/q235b_judge_dpo-s
status: torndown
timestamp: 2026-04-01T17:21:46+00:00
num_battles: 10063
num_wins: 5483
celo_rating: 1324.97
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/q235b_judge_dpo-step225-merged
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-q235b-judge-dpo-_47447_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/q235b_judge_dpo-step225-merged
model_size: 19B
ranking_group: single
us_pacific_date: 2026-03-29
win_ratio: 0.5448673357845573
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</think>', '<|assistant|>', '####', '<|user|>', '</s>', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-judge-dpo-47447-v1-uploader
Waiting for job on chaiml-q235b-judge-dpo-47447-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-judge-dpo-74524-v1-uploader
Waiting for job on chaiml-q235b-judge-dpo-74524-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v28-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v28-uploader to finish
chaiml-q235b-judge-dpo-74524-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-74524-v1-uploader: Checking if ChaiML/q235b_judge_dpo-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-74524-v1-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo-step450-merged...
chaiml-kimid-v9-opusdv-23365-v28-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v28-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v9-opusdv-23365-v28-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v28-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
chaiml-q235b-judge-dpo-47447-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-47447-v1-uploader: Checking if ChaiML/q235b_judge_dpo-step225-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-47447-v1-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo-step225-merged...
2026-03-29T06:03:23.038237+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:03:26.854797+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:03:31.034313+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:04:23.257752+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:04:27.118841+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:04:31.227818+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:05:23.440930+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
chaiml-kimid-v9-opusdv-23365-v28-uploader: Downloaded in 130.739s
chaiml-kimid-v9-opusdv-23365-v28-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 131.295s
chaiml-kimid-v9-opusdv-23365-v28-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
2026-03-29T06:05:27.297713+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:05:31.405497+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00027-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: Downloaded in 145.339s
chaiml-q235b-judge-dpo-74524-v1-uploader: Applying quantization...
chaiml-q235b-judge-dpo-47447-v1-uploader: Downloaded in 152.822s
chaiml-q235b-judge-dpo-47447-v1-uploader: Applying quantization...
chaiml-q235b-judge-dpo-47447-v1-uploader: [33;1m2026-03-28 23:05:38 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead![0m
chaiml-q235b-judge-dpo-74524-v1-uploader: [33;1m2026-03-28 23:05:47 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead![0m
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: [38;20m2026-03-28 23:05:57 INFO base.py L366: using torch.bfloat16 for quantization tuning[0m
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: [38;20m2026-03-28 23:05:59 INFO base.py L366: using torch.bfloat16 for quantization tuning[0m
chaiml-q235b-judge-dpo-47447-v1-uploader: [38;20m2026-03-28 23:06:04 INFO base.py L1145: start to compute imatrix[0m
chaiml-q235b-judge-dpo-74524-v1-uploader: [38;20m2026-03-28 23:06:02 INFO base.py L1145: start to compute imatrix[0m
Job chaiml-kimid-v9-opusdv-23365-v28-uploader completed after 221.66s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v28-uploader
Pipeline stage VLLMUploader completed in 222.68s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.90s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v28
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v28 to be ready
2026-03-29T06:06:23.633647+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:06:27.478228+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:06:31.578665+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/local/lib/python3.12/dist-packages/torch/backends/cuda/__init__.py:131: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
chaiml-q235b-judge-dpo-47447-v1-uploader:   return torch._C._get_cublas_allow_tf32()
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1343             # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56  # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1343             # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56  # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2026-03-29T06:07:23.848495+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
chaiml-q235b-judge-dpo-74524-v1-uploader: [33;1m2026-03-28 23:07:22 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0[0m
2026-03-29T06:07:27.682943+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader: [33;1m2026-03-28 23:07:28 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0[0m
2026-03-29T06:07:31.823942+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:08:24.053635+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:08:28.102825+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:08:32.047462+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:09:24.283904+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:09:28.315332+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:09:32.258231+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:10:24.522036+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:10:28.526745+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:10:32.460413+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:11:27.977466+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:11:30.774942+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:11:32.680907+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:12:28.201656+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:12:31.023368+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:12:33.184812+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:13:28.415058+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:13:31.245977+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:13:33.419486+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:14:28.676694+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:14:31.470805+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:14:33.649089+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:15:28.906314+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:15:31.686043+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:15:33.872461+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:16:29.120382+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:16:31.905754+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:16:34.136359+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:17:29.358874+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:17:32.124110+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:17:34.354074+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:18:29.579286+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:18:32.397807+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:18:34.592154+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:19:29.822841+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:19:32.602271+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:19:34.817865+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:20:30.054335+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:20:32.820368+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:20:35.033218+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:21:30.267759+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:21:33.043900+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:21:35.267702+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:22:30.486199+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:22:33.260592+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:22:35.502445+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:23:30.757027+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:23:33.489627+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:23:35.756486+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:24:30.988452+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:24:33.709379+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:24:35.982508+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:25:31.247534+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:25:33.930691+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:25:36.196330+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:26:31.462499+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:26:34.147923+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:26:36.410875+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:27:31.689059+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:27:34.364794+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:27:36.635034+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:28:31.959783+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:28:34.595472+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:28:36.858766+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:29:32.177518+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:29:34.821589+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:29:37.081951+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:30:32.419027+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:30:35.448067+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:30:37.329707+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:31:32.688060+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:31:35.698615+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:31:37.584181+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:32:32.933200+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:32:35.949638+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:32:37.830800+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:33:33.170611+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:33:36.187224+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:33:38.072023+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:34:33.441055+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:34:36.442620+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:34:38.317240+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:35:33.690422+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:35:36.695218+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:35:38.557435+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:36:33.936534+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:36:36.927483+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:36:38.804419+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:37:34.239925+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:37:37.184699+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:37:39.054294+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:38:34.483829+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:38:37.441949+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:38:39.310799+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
Retrying (%r) after connection broken by '%r': %s
2026-03-29T06:39:34.793512+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:39:37.982487+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:39:39.569910+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:40:35.065723+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:40:38.225656+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:40:39.818020+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:41:35.367621+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:41:38.475203+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:41:40.072313+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:42:35.616905+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:42:38.745210+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:42:40.324256+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:43:35.923943+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:43:38.994933+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:43:40.577313+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:44:36.236089+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:44:39.236066+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:44:40.834358+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:45:36.474330+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:45:39.489940+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:45:41.089378+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
Tearing down inference service chaiml-kimid-v9-opusdv-23365-v28
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-kimid-v9-opusdv-23365-v28.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'40m\'}, \'creationTimestamp\': \'2026-03-29T06:06:19Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'istio.io/rev\': \'prod-canary\', \'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:istio.io/rev\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-03-29T06:06:19Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-03-29T06:06:19Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-03-29T06:06:20Z\'}], \'name\': \'chaiml-kimid-v9-opusdv-23365-v28\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'1300218390\', \'uid\': \'967ec698-62e9-4a11-9490-1203225fd949\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'8\', \'--gpu-memory-utilization\', \'0.9\', \'--max-model-len\', \'10240\', \'--max-num-batched-tokens\', \'10240\', \'--max-num-seqs\', \'64\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01\', \'--model-loader-extra-config\', \'{"distributed": true, "concurrency": 2}\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'8\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'1\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'CWZAGMHZXKZRFGJK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'cwoAeWzp46q4O0sTNXOEuZ1MvZzKEFlS9DtEhnTldKp\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://cwobject.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.17.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}, \'requests\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 5, \'minReplicas\': 0, \'priorityClassName\': \'chaiverse\', \'timeout\': 20, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'163Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-kimid-v9-opusdv-23365-v28-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Revision "chaiml-kimid-v9-opusdv-23365-v28-predictor-00001" failed with message: 0/276 nodes are available: 1 Insufficient cpu, 122 Insufficient nvidia.com/gpu, 147 node(s) didn\\\'t match Pod\\\'s node affinity/selector, 2 node(s) had untolerated taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, 5 node(s) were unschedulable. preemption: 0/276 nodes are available: 122 Insufficient nvidia.com/gpu, 154 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.53s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/.gitattributes from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/added_tokens.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/merges.txt from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00001-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00002-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00003-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00004-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00005-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00006-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00007-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00008-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00009-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00010-of-00027.safetensors from bucket guanaco-vllm-models
2026-03-29T06:46:36.725700+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00011-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00012-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00013-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00014-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00015-of-00027.safetensors from bucket guanaco-vllm-models
2026-03-29T06:46:40.115875+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00016-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00017-of-00027.safetensors from bucket guanaco-vllm-models
2026-03-29T06:46:41.338643+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00018-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00019-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00020-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00021-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00022-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00023-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00024-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00025-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00026-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00027-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/quantization_config.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/tokenizer_config.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/vocab.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 24.33s
Shutdown handler de-registered
DeploymentError('Timeout to start the InferenceService chaiml-kimid-v9-opusdv-23365-v28.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'40m\'}, \'creationTimestamp\': \'2026-03-29T06:06:19Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'istio.io/rev\': \'prod-canary\', \'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:istio.io/rev\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-03-29T06:06:19Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-03-29T06:06:19Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-03-29T06:06:20Z\'}], \'name\': \'chaiml-kimid-v9-opusdv-23365-v28\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'1300218390\', \'uid\': \'967ec698-62e9-4a11-9490-1203225fd949\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'8\', \'--gpu-memory-utilization\', \'0.9\', \'--max-model-len\', \'10240\', \'--max-num-batched-tokens\', \'10240\', \'--max-num-seqs\', \'64\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01\', \'--model-loader-extra-config\', \'{"distributed": true, "concurrency": 2}\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'8\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'1\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'CWZAGMHZXKZRFGJK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'cwoAeWzp46q4O0sTNXOEuZ1MvZzKEFlS9DtEhnTldKp\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://cwobject.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.17.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}, \'requests\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 5, \'minReplicas\': 0, \'priorityClassName\': \'chaiverse\', \'timeout\': 20, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'163Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-kimid-v9-opusdv-23365-v28-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Revision "chaiml-kimid-v9-opusdv-23365-v28-predictor-00001" failed with message: 0/276 nodes are available: 1 Insufficient cpu, 122 Insufficient nvidia.com/gpu, 147 node(s) didn\\\'t match Pod\\\'s node affinity/selector, 2 node(s) had untolerated taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, 5 node(s) were unschedulable. preemption: 0/276 nodes are available: 122 Insufficient nvidia.com/gpu, 154 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
chaiml-kimid-v9-opusdv_23365_v28 status is now failed due to DeploymentManager action
2026-03-29T06:47:37.016624+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:47:40.473392+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:48:37.229463+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:48:40.692101+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:49:37.432251+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:49:40.904695+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:50:37.711372+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:50:41.101557+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:51:38.040429+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:51:41.298256+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:52:38.241479+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:52:41.500337+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:53:38.447101+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:53:41.706732+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:54:38.741262+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:54:41.910669+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:55:39.046849+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:55:42.109321+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:56:39.248070+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:56:42.287927+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:57:39.437924+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:57:42.493011+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:58:39.724554+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:58:42.667139+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:59:40.881428+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:59:42.851708+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:00:41.064131+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:00:43.035418+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:01:41.252180+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:01:43.218661+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:02:41.444102+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:02:43.415820+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:03:41.628006+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:03:43.591640+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:04:41.815241+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:04:43.791397+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:05:42.015787+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:05:43.986999+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:06:42.337064+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:06:44.171723+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:07:42.514670+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:07:44.380812+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:08:42.726023+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:08:44.563026+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:09:43.026462+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:09:44.762249+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:10:43.231724+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:10:44.955944+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:11:43.431065+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:11:45.160193+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:12:43.639509+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:12:45.366328+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:13:43.845693+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:13:45.568078+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:14:44.054515+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:14:45.775784+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:15:46.014303+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:15:46.113604+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:16:46.196138+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:16:46.336376+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:17:46.783285+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:17:46.864847+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:18:47.023558+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:18:47.106358+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:19:47.211344+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:19:47.321585+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-74524-v1-uploader: Checking if ChaiML/q235b_judge_dpo-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-74524-v1-uploader: Creating repo ChaiML/q235b_judge_dpo-step450-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:20:14 (0:00:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+27 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 27 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 98
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:20:47.415273+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:20:47.535551+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader: Checking if ChaiML/q235b_judge_dpo-step225-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-47447-v1-uploader: Creating repo ChaiML/q235b_judge_dpo-step225-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:21:04 (0:00:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+27 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 27 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 98
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:21:14 (0:01:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 2/28 (1.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:21:47.608302+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:21:47.750527+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:22:04 (0:01:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 2/28 (1.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:22:14 (0:02:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:22:47.816871+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:22:47.962767+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:23:04 (0:02:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:23:14 (0:03:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:23:48.027404+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:23:48.182833+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:24:04 (0:03:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:24:14 (0:04:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 19/28 (86.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 9 | committing: 0 | waiting: 117
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:24:48.249405+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:24:48.413216+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:25:04 (0:04:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 26/28 (121.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 2 | committing: 0 | waiting: 124
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:25:14 (0:05:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:25:48.473587+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:25:48.640651+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-74524-v1-uploader:                              Processed model ChaiML/q235b_judge_dpo-step450-merged in 4958.809s
chaiml-q235b-judge-dpo-74524-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/vocab.json
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model.safetensors.index.json
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/tokenizer.json
chaiml-q235b-judge-dpo-47447-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:26:04 (0:05:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00027-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader:                              Processed model ChaiML/q235b_judge_dpo-step225-merged in 4995.371s
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00001-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00024-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00022-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00018-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00012-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-judge-dpo-47447-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-judge-dpo-47447-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-judge-dpo-47447-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-judge-dpo-47447-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-judge-dpo-47447-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/merges.txt
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/special_tokens_map.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/quantization_config.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/chat_template.jinja
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/tokenizer_config.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/config.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/generation_config.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/added_tokens.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/vocab.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/tokenizer.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model.safetensors.index.json
Job chaiml-q235b-judge-dpo-74524-v1-uploader completed after 5050.59s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-74524-v1-uploader
Pipeline stage VLLMUploader completed in 5051.48s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.51s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-74524-v1
Waiting for inference service chaiml-q235b-judge-dpo-74524-v1 to be ready
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00027-of-00027.safetensors
2026-03-29T07:26:48.727784+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:26:48.933534+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00012-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00018-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00001-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00023-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00024-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00022-of-00027.safetensors
Job chaiml-q235b-judge-dpo-47447-v1-uploader completed after 5088.79s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-47447-v1-uploader
Pipeline stage VLLMUploader completed in 5089.62s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.51s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-47447-v1
Waiting for inference service chaiml-q235b-judge-dpo-47447-v1 to be ready
2026-03-29T07:27:48.981936+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:27:49.192395+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:28:49.220485+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:28:49.412513+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:29:49.454243+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:29:49.653150+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:30:49.690081+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:30:49.873132+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:31:49.908845+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:31:50.137720+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:32:50.253620+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:32:50.474329+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:33:50.471239+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:33:50.702950+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:34:50.677920+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:34:50.931558+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:35:50.941554+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:35:51.268339+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:36:51.183435+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:36:51.505395+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:37:51.496500+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:37:51.756728+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:38:51.763322+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:38:52.175499+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:39:52.075028+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:39:52.415051+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:40:52.336446+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:40:52.661362+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:41:52.642270+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:41:52.903024+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:42:52.965482+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:42:53.244417+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:43:53.192758+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:43:53.474933+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:44:53.409393+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:44:53.684761+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:45:54.050372+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:45:54.185455+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:46:54.359761+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:46:54.456970+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:47:54.588329+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:47:54.715037+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:48:54.882253+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:48:55.079845+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:49:55.120823+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:49:55.333180+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:50:55.356067+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:50:55.567729+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:51:55.594933+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:51:55.829081+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:52:55.903470+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:52:56.096274+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:53:56.147045+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:53:56.350570+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:54:56.385045+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:54:56.662110+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
Inference service chaiml-q235b-judge-dpo-74524-v1 ready after 1722.3143730163574s
Pipeline stage VLLMDeployer completed in 1732.83s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.8938069343566895s
Received healthy response to inference request in 1.4668560028076172s
Received healthy response to inference request in 1.549198865890503s
Received healthy response to inference request in 1.4455840587615967s
Received healthy response to inference request in 1.4595539569854736s
Received healthy response to inference request in 1.5154740810394287s
Received healthy response to inference request in 1.4504642486572266s
Received healthy response to inference request in 1.5450689792633057s
Received healthy response to inference request in 1.4582180976867676s
Received healthy response to inference request in 1.4737815856933594s
Received healthy response to inference request in 1.5705575942993164s
Received healthy response to inference request in 1.5037624835968018s
2026-03-29T07:55:56.727368+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:55:56.918381+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
Received healthy response to inference request in 1.5411100387573242s
Received healthy response to inference request in 1.4316432476043701s
Received healthy response to inference request in 1.409142017364502s
Received healthy response to inference request in 1.4927904605865479s
Received healthy response to inference request in 1.4638051986694336s
Received healthy response to inference request in 1.61080002784729s
Received healthy response to inference request in 1.5494799613952637s
Received healthy response to inference request in 1.4419567584991455s
Received healthy response to inference request in 1.5698966979980469s
Received healthy response to inference request in 1.5282354354858398s
Received healthy response to inference request in 1.6968274116516113s
Received healthy response to inference request in 1.4601237773895264s
Received healthy response to inference request in 1.6880383491516113s
Received healthy response to inference request in 1.4570155143737793s
Received healthy response to inference request in 1.5159761905670166s
Received healthy response to inference request in 1.4480791091918945s
Received healthy response to inference request in 1.4532725811004639s
Received healthy response to inference request in 1.529050350189209s
30 requests
0 failed requests
5th percentile: 1.436284327507019
10th percentile: 1.4452213287353515
20th percentile: 1.4527109146118165
30th percentile: 1.4591531991958617
40th percentile: 1.4656356811523437
50th percentile: 1.4982764720916748
60th percentile: 1.5208798885345458
70th percentile: 1.5422977209091187
80th percentile: 1.5535633087158203
90th percentile: 1.6185238599777223
95th percentile: 1.6928723335266114
99th percentile: 3.2566828727722186
mean time: 1.5873190005620321
Pipeline stage StressChecker completed in 59.91s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.44s
Shutdown handler de-registered
chaiml-q235b-judge-dpo-_74524_v1 status is now deployed due to DeploymentManager action
2026-03-29T07:56:56.998826+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:57:57.169138+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:58:57.367082+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
Inference service chaiml-q235b-judge-dpo-47447-v1 ready after 1905.898277759552s
Pipeline stage VLLMDeployer completed in 1906.85s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.697683811187744s
Received healthy response to inference request in 1.7461340427398682s
Received healthy response to inference request in 1.4584314823150635s
Received healthy response to inference request in 1.4299967288970947s
Received healthy response to inference request in 1.4548265933990479s
Received healthy response to inference request in 1.512843370437622s
Received healthy response to inference request in 1.450890064239502s
Received healthy response to inference request in 1.513659954071045s
Received healthy response to inference request in 1.4425673484802246s
Received healthy response to inference request in 1.4487767219543457s
Received healthy response to inference request in 1.4622457027435303s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T07:59:57.512446+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T08:00:57.680836+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T08:01:57.876067+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T08:02:58.043824+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T08:03:58.198061+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T08:04:58.699494+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
30 requests
19 failed requests
5th percentile: 1.445361566543579
10th percentile: 1.4506787300109862
20th percentile: 1.461482858657837
30th percentile: 1.676391816139221
40th percentile: 20.256861209869385
50th percentile: 20.264703273773193
60th percentile: 20.27636375427246
70th percentile: 20.28540518283844
80th percentile: 20.304516506195068
90th percentile: 20.33300051689148
95th percentile: 20.576263737678527
99th percentile: 20.775891892910003
mean time: 13.502329572041829
%s, retrying in %s seconds...
2026-03-29T08:05:58.899369+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T08:06:59.111506+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T08:07:59.579454+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 13.465453386306763s
Received healthy response to inference request in 1.5440959930419922s
Received healthy response to inference request in 1.4566221237182617s
Received healthy response to inference request in 1.4797472953796387s
Received healthy response to inference request in 1.5612983703613281s
Received healthy response to inference request in 1.6959171295166016s
Received healthy response to inference request in 1.4275312423706055s
Received healthy response to inference request in 1.5036044120788574s
Received healthy response to inference request in 1.6841459274291992s
Received healthy response to inference request in 1.5788073539733887s
Received healthy response to inference request in 1.5764353275299072s
Received healthy response to inference request in 1.4676008224487305s
Received healthy response to inference request in 1.439382791519165s
Received healthy response to inference request in 1.4582104682922363s
Received healthy response to inference request in 1.4402029514312744s
Received healthy response to inference request in 1.4649534225463867s
Received healthy response to inference request in 1.5429370403289795s
Received healthy response to inference request in 1.7635536193847656s
Received healthy response to inference request in 1.472869873046875s
Received healthy response to inference request in 1.5196199417114258s
Received healthy response to inference request in 1.514451026916504s
2026-03-29T08:08:59.748554+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
Received healthy response to inference request in 1.4988172054290771s
Received healthy response to inference request in 1.4746594429016113s
30 requests
7 failed requests
5th percentile: 1.4397518634796143
10th percentile: 1.454980206489563
20th percentile: 1.4670713424682618
30th percentile: 1.4782209396362305
40th percentile: 1.5101123809814454
50th percentile: 1.5435165166854858
60th percentile: 1.5773841381072997
70th percentile: 1.7162080764770506
80th percentile: 20.265116453170776
90th percentile: 20.277255630493165
95th percentile: 20.289134752750396
99th percentile: 20.302084383964537
mean time: 6.300024620691935
%s, retrying in %s seconds...
Received healthy response to inference request in 1.4421210289001465s
Received healthy response to inference request in 1.5504100322723389s
Received healthy response to inference request in 1.5096700191497803s
Received healthy response to inference request in 1.547346591949463s
Received healthy response to inference request in 1.682154655456543s
Received healthy response to inference request in 1.4696261882781982s
Received healthy response to inference request in 1.4300332069396973s
Received healthy response to inference request in 1.4446535110473633s
Received healthy response to inference request in 1.7060141563415527s
Received healthy response to inference request in 1.5463893413543701s
Received healthy response to inference request in 1.46480131149292s
Received healthy response to inference request in 1.5111234188079834s
Received healthy response to inference request in 1.545226812362671s
Received healthy response to inference request in 1.4467096328735352s
Received healthy response to inference request in 1.4446475505828857s
Received healthy response to inference request in 1.4882962703704834s
Received healthy response to inference request in 1.4591026306152344s
Received healthy response to inference request in 1.460066556930542s
Received healthy response to inference request in 1.4511566162109375s
Received healthy response to inference request in 1.6007134914398193s
Received healthy response to inference request in 1.4507050514221191s
Received healthy response to inference request in 1.4550154209136963s
Received healthy response to inference request in 1.4445140361785889s
Received healthy response to inference request in 1.4486608505249023s
Received healthy response to inference request in 1.4441604614257812s
Received healthy response to inference request in 1.706979513168335s
Received healthy response to inference request in 1.4391191005706787s
Received healthy response to inference request in 1.4615154266357422s
Received healthy response to inference request in 1.5325863361358643s
Received healthy response to inference request in 1.4810986518859863s
30 requests
0 failed requests
5th percentile: 1.4404699683189393
10th percentile: 1.4439565181732177
20th percentile: 1.4446523189544678
30th percentile: 1.450091791152954
40th percentile: 1.457467746734619
50th percentile: 1.463158369064331
60th percentile: 1.483977699279785
70th percentile: 1.5175622940063476
80th percentile: 1.5465807914733887
90th percentile: 1.6088576078414918
95th percentile: 1.6952773809432982
99th percentile: 1.7066995596885681
mean time: 1.5021539290746053
Pipeline stage StressChecker completed in 654.02s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.09s
Shutdown handler de-registered
chaiml-q235b-judge-dpo-_47447_v1 status is now deployed due to DeploymentManager action
chaiml-q235b-judge-dpo-_47447_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-q235b-judge-dpo-_47447_v1 status is now torndown due to DeploymentManager action