chaiml-q235b-judge-dpo-_74524

developer_uid: rirv938
submission_id: chaiml-q235b-judge-dpo-_74524_v1
model_name: chaiml-q235b-judge-dpo-_74524_v1
model_group: ChaiML/q235b_judge_dpo-s
status: torndown
timestamp: 2026-04-01T20:41:12+00:00
num_battles: 10026
num_wins: 5537
celo_rating: 1331.73
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/q235b_judge_dpo-step450-merged
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-q235b-judge-dpo-_74524_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/q235b_judge_dpo-step450-merged
model_size: 19B
ranking_group: single
us_pacific_date: 2026-03-29
win_ratio: 0.5522641133054059
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</think>', '####', '<|user|>', '</s>', '<|assistant|>', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-judge-dpo-74524-v1-uploader
Waiting for job on chaiml-q235b-judge-dpo-74524-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v28-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v28-uploader to finish
chaiml-q235b-judge-dpo-74524-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-74524-v1-uploader: Checking if ChaiML/q235b_judge_dpo-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-74524-v1-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo-step450-merged...
chaiml-kimid-v9-opusdv-23365-v28-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v28-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v9-opusdv-23365-v28-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v28-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
chaiml-q235b-judge-dpo-47447-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-47447-v1-uploader: Checking if ChaiML/q235b_judge_dpo-step225-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-47447-v1-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo-step225-merged...
2026-03-29T06:03:23.038237+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:03:26.854797+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:03:31.034313+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:04:23.257752+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:04:27.118841+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:04:31.227818+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:05:23.440930+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
chaiml-kimid-v9-opusdv-23365-v28-uploader: Downloaded in 130.739s
chaiml-kimid-v9-opusdv-23365-v28-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 131.295s
chaiml-kimid-v9-opusdv-23365-v28-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v28-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v9-opusdv-23365-v28-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
2026-03-29T06:05:27.297713+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:05:31.405497+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00027-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: Downloaded in 145.339s
chaiml-q235b-judge-dpo-74524-v1-uploader: Applying quantization...
chaiml-q235b-judge-dpo-47447-v1-uploader: Downloaded in 152.822s
chaiml-q235b-judge-dpo-47447-v1-uploader: Applying quantization...
chaiml-q235b-judge-dpo-47447-v1-uploader: [33;1m2026-03-28 23:05:38 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead![0m
chaiml-q235b-judge-dpo-74524-v1-uploader: [33;1m2026-03-28 23:05:47 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead![0m
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: [38;20m2026-03-28 23:05:57 INFO base.py L366: using torch.bfloat16 for quantization tuning[0m
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v28-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: [38;20m2026-03-28 23:05:59 INFO base.py L366: using torch.bfloat16 for quantization tuning[0m
chaiml-q235b-judge-dpo-47447-v1-uploader: [38;20m2026-03-28 23:06:04 INFO base.py L1145: start to compute imatrix[0m
chaiml-q235b-judge-dpo-74524-v1-uploader: [38;20m2026-03-28 23:06:02 INFO base.py L1145: start to compute imatrix[0m
Job chaiml-kimid-v9-opusdv-23365-v28-uploader completed after 221.66s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v28-uploader
Pipeline stage VLLMUploader completed in 222.68s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.90s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v28
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v28 to be ready
2026-03-29T06:06:23.633647+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:06:27.478228+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:06:31.578665+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/local/lib/python3.12/dist-packages/torch/backends/cuda/__init__.py:131: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
chaiml-q235b-judge-dpo-47447-v1-uploader:   return torch._C._get_cublas_allow_tf32()
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1343             # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:01.577000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56  # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-74524-v1-uploader: W0328 23:07:04.507000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1343             # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:07.250000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56  # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-47447-v1-uploader: W0328 23:07:10.102000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2026-03-29T06:07:23.848495+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
chaiml-q235b-judge-dpo-74524-v1-uploader: [33;1m2026-03-28 23:07:22 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0[0m
2026-03-29T06:07:27.682943+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader: [33;1m2026-03-28 23:07:28 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0[0m
2026-03-29T06:07:31.823942+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:08:24.053635+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:08:28.102825+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:08:32.047462+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:09:24.283904+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:09:28.315332+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:09:32.258231+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:10:24.522036+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:10:28.526745+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:10:32.460413+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:11:27.977466+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:11:30.774942+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:11:32.680907+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:12:28.201656+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:12:31.023368+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:12:33.184812+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:13:28.415058+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:13:31.245977+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:13:33.419486+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:14:28.676694+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:14:31.470805+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:14:33.649089+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:15:28.906314+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:15:31.686043+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:15:33.872461+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:16:29.120382+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:16:31.905754+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:16:34.136359+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:17:29.358874+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:17:32.124110+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:17:34.354074+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:18:29.579286+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:18:32.397807+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:18:34.592154+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:19:29.822841+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:19:32.602271+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:19:34.817865+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:20:30.054335+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:20:32.820368+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:20:35.033218+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:21:30.267759+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:21:33.043900+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:21:35.267702+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:22:30.486199+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:22:33.260592+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:22:35.502445+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:23:30.757027+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:23:33.489627+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:23:35.756486+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:24:30.988452+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:24:33.709379+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:24:35.982508+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:25:31.247534+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:25:33.930691+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:25:36.196330+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:26:31.462499+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:26:34.147923+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:26:36.410875+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:27:31.689059+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:27:34.364794+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:27:36.635034+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:28:31.959783+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:28:34.595472+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:28:36.858766+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:29:32.177518+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:29:34.821589+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:29:37.081951+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:30:32.419027+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:30:35.448067+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:30:37.329707+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:31:32.688060+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:31:35.698615+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:31:37.584181+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:32:32.933200+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:32:35.949638+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:32:37.830800+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:33:33.170611+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:33:36.187224+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:33:38.072023+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:34:33.441055+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:34:36.442620+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:34:38.317240+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:35:33.690422+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:35:36.695218+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:35:38.557435+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:36:33.936534+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:36:36.927483+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:36:38.804419+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:37:34.239925+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:37:37.184699+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:37:39.054294+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:38:34.483829+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:38:37.441949+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:38:39.310799+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
Retrying (%r) after connection broken by '%r': %s
2026-03-29T06:39:34.793512+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:39:37.982487+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:39:39.569910+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:40:35.065723+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:40:38.225656+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:40:39.818020+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:41:35.367621+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:41:38.475203+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:41:40.072313+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:42:35.616905+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:42:38.745210+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:42:40.324256+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:43:35.923943+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:43:38.994933+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:43:40.577313+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:44:36.236089+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:44:39.236066+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:44:40.834358+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
2026-03-29T06:45:36.474330+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:45:39.489940+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:45:41.089378+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
Tearing down inference service chaiml-kimid-v9-opusdv-23365-v28
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-kimid-v9-opusdv-23365-v28.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'40m\'}, \'creationTimestamp\': \'2026-03-29T06:06:19Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'istio.io/rev\': \'prod-canary\', \'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:istio.io/rev\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-03-29T06:06:19Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-03-29T06:06:19Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-03-29T06:06:20Z\'}], \'name\': \'chaiml-kimid-v9-opusdv-23365-v28\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'1300218390\', \'uid\': \'967ec698-62e9-4a11-9490-1203225fd949\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'8\', \'--gpu-memory-utilization\', \'0.9\', \'--max-model-len\', \'10240\', \'--max-num-batched-tokens\', \'10240\', \'--max-num-seqs\', \'64\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01\', \'--model-loader-extra-config\', \'{"distributed": true, "concurrency": 2}\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'8\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'1\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'CWZAGMHZXKZRFGJK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'cwoAeWzp46q4O0sTNXOEuZ1MvZzKEFlS9DtEhnTldKp\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://cwobject.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.17.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}, \'requests\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 5, \'minReplicas\': 0, \'priorityClassName\': \'chaiverse\', \'timeout\': 20, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'163Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-kimid-v9-opusdv-23365-v28-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Revision "chaiml-kimid-v9-opusdv-23365-v28-predictor-00001" failed with message: 0/276 nodes are available: 1 Insufficient cpu, 122 Insufficient nvidia.com/gpu, 147 node(s) didn\\\'t match Pod\\\'s node affinity/selector, 2 node(s) had untolerated taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, 5 node(s) were unschedulable. preemption: 0/276 nodes are available: 122 Insufficient nvidia.com/gpu, 154 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.53s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/.gitattributes from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/added_tokens.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/merges.txt from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00001-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00002-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00003-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00004-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00005-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00006-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00007-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00008-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00009-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00010-of-00027.safetensors from bucket guanaco-vllm-models
2026-03-29T06:46:36.725700+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00011-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00012-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00013-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00014-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00015-of-00027.safetensors from bucket guanaco-vllm-models
2026-03-29T06:46:40.115875+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00016-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00017-of-00027.safetensors from bucket guanaco-vllm-models
2026-03-29T06:46:41.338643+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v28
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00018-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00019-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00020-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00021-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00022-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00023-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00024-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00025-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00026-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model-00027-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/quantization_config.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/tokenizer_config.json from bucket guanaco-vllm-models
Deleting key chaiml-kimid-v9-opusdv-23365-v28/default/vocab.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 24.33s
Shutdown handler de-registered
DeploymentError('Timeout to start the InferenceService chaiml-kimid-v9-opusdv-23365-v28.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'40m\'}, \'creationTimestamp\': \'2026-03-29T06:06:19Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'istio.io/rev\': \'prod-canary\', \'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:istio.io/rev\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-03-29T06:06:19Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-03-29T06:06:19Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-03-29T06:06:20Z\'}], \'name\': \'chaiml-kimid-v9-opusdv-23365-v28\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'1300218390\', \'uid\': \'967ec698-62e9-4a11-9490-1203225fd949\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v28/default\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'8\', \'--gpu-memory-utilization\', \'0.9\', \'--max-model-len\', \'10240\', \'--max-num-batched-tokens\', \'10240\', \'--max-num-seqs\', \'64\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01\', \'--model-loader-extra-config\', \'{"distributed": true, "concurrency": 2}\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'8\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'1\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'CWZAGMHZXKZRFGJK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'cwoAeWzp46q4O0sTNXOEuZ1MvZzKEFlS9DtEhnTldKp\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://cwobject.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.17.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}, \'requests\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 5, \'minReplicas\': 0, \'priorityClassName\': \'chaiverse\', \'timeout\': 20, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'163Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-kimid-v9-opusdv-23365-v28-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Revision "chaiml-kimid-v9-opusdv-23365-v28-predictor-00001" failed with message: 0/276 nodes are available: 1 Insufficient cpu, 122 Insufficient nvidia.com/gpu, 147 node(s) didn\\\'t match Pod\\\'s node affinity/selector, 2 node(s) had untolerated taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, 5 node(s) were unschedulable. preemption: 0/276 nodes are available: 122 Insufficient nvidia.com/gpu, 154 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'message\': \'Configuration "chaiml-kimid-v9-opusdv-23365-v28-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-03-29T06:06:20Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
chaiml-kimid-v9-opusdv_23365_v28 status is now failed due to DeploymentManager action
2026-03-29T06:47:37.016624+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:47:40.473392+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:48:37.229463+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:48:40.692101+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:49:37.432251+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:49:40.904695+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:50:37.711372+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:50:41.101557+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:51:38.040429+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:51:41.298256+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:52:38.241479+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:52:41.500337+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:53:38.447101+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:53:41.706732+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:54:38.741262+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:54:41.910669+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:55:39.046849+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:55:42.109321+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:56:39.248070+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:56:42.287927+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:57:39.437924+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:57:42.493011+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:58:39.724554+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:58:42.667139+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T06:59:40.881428+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T06:59:42.851708+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:00:41.064131+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:00:43.035418+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:01:41.252180+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:01:43.218661+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:02:41.444102+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:02:43.415820+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:03:41.628006+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:03:43.591640+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:04:41.815241+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:04:43.791397+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:05:42.015787+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:05:43.986999+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:06:42.337064+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:06:44.171723+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:07:42.514670+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:07:44.380812+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:08:42.726023+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:08:44.563026+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:09:43.026462+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:09:44.762249+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:10:43.231724+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:10:44.955944+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:11:43.431065+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:11:45.160193+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:12:43.639509+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:12:45.366328+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:13:43.845693+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:13:45.568078+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:14:44.054515+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:14:45.775784+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:15:46.014303+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:15:46.113604+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:16:46.196138+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:16:46.336376+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:17:46.783285+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:17:46.864847+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:18:47.023558+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:18:47.106358+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:19:47.211344+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:19:47.321585+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-74524-v1-uploader: Checking if ChaiML/q235b_judge_dpo-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-74524-v1-uploader: Creating repo ChaiML/q235b_judge_dpo-step450-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:20:14 (0:00:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+27 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 27 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 98
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:20:47.415273+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:20:47.535551+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader: Checking if ChaiML/q235b_judge_dpo-step225-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-47447-v1-uploader: Creating repo ChaiML/q235b_judge_dpo-step225-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:21:04 (0:00:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+27 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 27 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 98
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:21:14 (0:01:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 2/28 (1.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:21:47.608302+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:21:47.750527+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:22:04 (0:01:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 2/28 (1.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:22:14 (0:02:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:22:47.816871+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:22:47.962767+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:23:04 (0:02:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:23:14 (0:03:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:23:48.027404+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:23:48.182833+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:24:04 (0:03:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:24:14 (0:04:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 19/28 (86.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 9 | committing: 0 | waiting: 117
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:24:48.249405+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:24:48.413216+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:25:04 (0:04:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 26/28 (121.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 2 | committing: 0 | waiting: 124
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------- 2026-03-29 00:25:14 (0:05:00) ----------
chaiml-q235b-judge-dpo-74524-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-74524-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-q235b-judge-dpo-74524-v1-uploader: ---------------------------------------------------
2026-03-29T07:25:48.473587+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:25:48.640651+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-74524-v1-uploader:                              Processed model ChaiML/q235b_judge_dpo-step450-merged in 4958.809s
chaiml-q235b-judge-dpo-74524-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/vocab.json
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model.safetensors.index.json
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/tokenizer.json
chaiml-q235b-judge-dpo-47447-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------- 2026-03-29 00:26:04 (0:05:00) ----------
chaiml-q235b-judge-dpo-47447-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-47447-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-q235b-judge-dpo-47447-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00027-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader:                              Processed model ChaiML/q235b_judge_dpo-step225-merged in 4995.371s
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00001-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00024-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00022-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00018-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00012-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v1/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-judge-dpo-47447-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-judge-dpo-47447-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-judge-dpo-47447-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-judge-dpo-47447-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-judge-dpo-47447-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-judge-dpo-47447-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/merges.txt
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/special_tokens_map.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/quantization_config.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/chat_template.jinja
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/tokenizer_config.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/config.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/generation_config.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/added_tokens.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/vocab.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/tokenizer.json
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model.safetensors.index.json
Job chaiml-q235b-judge-dpo-74524-v1-uploader completed after 5050.59s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-74524-v1-uploader
Pipeline stage VLLMUploader completed in 5051.48s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.51s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-74524-v1
Waiting for inference service chaiml-q235b-judge-dpo-74524-v1 to be ready
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00027-of-00027.safetensors
2026-03-29T07:26:48.727784+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:26:48.933534+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00012-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00018-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00001-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00023-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00024-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v1/default/model-00022-of-00027.safetensors
Job chaiml-q235b-judge-dpo-47447-v1-uploader completed after 5088.79s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-47447-v1-uploader
Pipeline stage VLLMUploader completed in 5089.62s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.51s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-47447-v1
Waiting for inference service chaiml-q235b-judge-dpo-47447-v1 to be ready
2026-03-29T07:27:48.981936+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:27:49.192395+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:28:49.220485+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:28:49.412513+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:29:49.454243+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:29:49.653150+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:30:49.690081+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:30:49.873132+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:31:49.908845+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:31:50.137720+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:32:50.253620+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:32:50.474329+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:33:50.471239+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:33:50.702950+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:34:50.677920+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:34:50.931558+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:35:50.941554+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:35:51.268339+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:36:51.183435+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:36:51.505395+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:37:51.496500+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:37:51.756728+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:38:51.763322+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:38:52.175499+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:39:52.075028+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:39:52.415051+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:40:52.336446+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:40:52.661362+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:41:52.642270+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:41:52.903024+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:42:52.965482+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:42:53.244417+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:43:53.192758+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:43:53.474933+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:44:53.409393+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:44:53.684761+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:45:54.050372+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:45:54.185455+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:46:54.359761+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:46:54.456970+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:47:54.588329+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:47:54.715037+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:48:54.882253+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:48:55.079845+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:49:55.120823+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:49:55.333180+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:50:55.356067+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:50:55.567729+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:51:55.594933+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:51:55.829081+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:52:55.903470+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:52:56.096274+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:53:56.147045+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:53:56.350570+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
2026-03-29T07:54:56.385045+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:54:56.662110+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
Inference service chaiml-q235b-judge-dpo-74524-v1 ready after 1722.3143730163574s
Pipeline stage VLLMDeployer completed in 1732.83s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.8938069343566895s
Received healthy response to inference request in 1.4668560028076172s
Received healthy response to inference request in 1.549198865890503s
Received healthy response to inference request in 1.4455840587615967s
Received healthy response to inference request in 1.4595539569854736s
Received healthy response to inference request in 1.5154740810394287s
Received healthy response to inference request in 1.4504642486572266s
Received healthy response to inference request in 1.5450689792633057s
Received healthy response to inference request in 1.4582180976867676s
Received healthy response to inference request in 1.4737815856933594s
Received healthy response to inference request in 1.5705575942993164s
Received healthy response to inference request in 1.5037624835968018s
2026-03-29T07:55:56.727368+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v1
2026-03-29T07:55:56.918381+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v1
Received healthy response to inference request in 1.5411100387573242s
Received healthy response to inference request in 1.4316432476043701s
Received healthy response to inference request in 1.409142017364502s
Received healthy response to inference request in 1.4927904605865479s
Received healthy response to inference request in 1.4638051986694336s
Received healthy response to inference request in 1.61080002784729s
Received healthy response to inference request in 1.5494799613952637s
Received healthy response to inference request in 1.4419567584991455s
Received healthy response to inference request in 1.5698966979980469s
Received healthy response to inference request in 1.5282354354858398s
Received healthy response to inference request in 1.6968274116516113s
Received healthy response to inference request in 1.4601237773895264s
Received healthy response to inference request in 1.6880383491516113s
Received healthy response to inference request in 1.4570155143737793s
Received healthy response to inference request in 1.5159761905670166s
Received healthy response to inference request in 1.4480791091918945s
Received healthy response to inference request in 1.4532725811004639s
Received healthy response to inference request in 1.529050350189209s
30 requests
0 failed requests
5th percentile: 1.436284327507019
10th percentile: 1.4452213287353515
20th percentile: 1.4527109146118165
30th percentile: 1.4591531991958617
40th percentile: 1.4656356811523437
50th percentile: 1.4982764720916748
60th percentile: 1.5208798885345458
70th percentile: 1.5422977209091187
80th percentile: 1.5535633087158203
90th percentile: 1.6185238599777223
95th percentile: 1.6928723335266114
99th percentile: 3.2566828727722186
mean time: 1.5873190005620321
Pipeline stage StressChecker completed in 59.91s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.44s
Shutdown handler de-registered
chaiml-q235b-judge-dpo-_74524_v1 status is now deployed due to DeploymentManager action
chaiml-q235b-judge-dpo-_74524_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-q235b-judge-dpo-_74524_v1 status is now torndown due to DeploymentManager action