Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-judge-dpo-18429-v1-uploader
Waiting for job on chaiml-q235b-judge-dpo-18429-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-judge-dpo-10887-v1-uploader
Waiting for job on chaiml-q235b-judge-dpo-10887-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v31-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v31-uploader to finish
chaiml-q235b-judge-dpo-18429-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-18429-v1-uploader: Checking if ChaiML/q235b_judge_dpo_lr1-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-18429-v1-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo_lr1-step450-merged...
chaiml-q235b-judge-dpo-10887-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-10887-v1-uploader: Checking if ChaiML/q235b_judge_dpo_lr1-step225-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-10887-v1-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo_lr1-step225-merged...
chaiml-kimid-v9-opusdv-23365-v31-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v31-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
2026-03-30T05:54:38.280976+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
chaiml-kimid-v9-opusdv-23365-v31-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v31-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
2026-03-30T05:54:42.214109+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T05:54:46.577993+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v31
2026-03-30T05:55:38.548813+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T05:55:42.402116+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T05:55:46.771576+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v31
2026-03-30T05:56:38.957872+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
chaiml-kimid-v9-opusdv-23365-v31-uploader: Downloaded in 116.123s
chaiml-kimid-v9-opusdv-23365-v31-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 116.757s
chaiml-kimid-v9-opusdv-23365-v31-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-23365-v31-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v31-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v9-opusdv-23365-v31-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v9-opusdv-23365-v31-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v9-opusdv-23365-v31-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v31-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v31-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v31-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v31-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v31-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v31-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v31-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v31-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v9-opusdv-23365-v31-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v31-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
2026-03-30T05:56:42.600612+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-kimid-v9-opusdv-23365-v31-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v9-opusdv-23365-v31-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-23365-v31-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/added_tokens.json
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/generation_config.json
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/quantization_config.json
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/special_tokens_map.json
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/tokenizer_config.json
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/.gitattributes
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/config.json
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/merges.txt
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/chat_template.jinja
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/vocab.json
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/tokenizer.json
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model.safetensors.index.json
2026-03-30T05:56:46.976345+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v31
chaiml-q235b-judge-dpo-18429-v1-uploader: Downloaded in 158.260s
chaiml-q235b-judge-dpo-18429-v1-uploader: Applying quantization...
chaiml-q235b-judge-dpo-18429-v1-uploader: [33;1m2026-03-29 22:56:57 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead![0m
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: Downloaded in 158.622s
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: Applying quantization...
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00022-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: [33;1m2026-03-29 22:57:01 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead![0m
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v31-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v31/default/model-00021-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-23365-v31-uploader completed after 211.86s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v31-uploader
Pipeline stage VLLMUploader completed in 212.88s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.28s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v31
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v31 to be ready
2026-03-30T05:57:39.189057+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T05:57:42.873323+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T05:57:47.189529+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v31
chaiml-q235b-judge-dpo-18429-v1-uploader: [38;20m2026-03-29 22:57:47 INFO base.py L366: using torch.bfloat16 for quantization tuning[0m
chaiml-q235b-judge-dpo-18429-v1-uploader: [38;20m2026-03-29 22:57:51 INFO base.py L1145: start to compute imatrix[0m
chaiml-q235b-judge-dpo-10887-v1-uploader: [38;20m2026-03-29 22:57:51 INFO base.py L366: using torch.bfloat16 for quantization tuning[0m
chaiml-q235b-judge-dpo-10887-v1-uploader: [38;20m2026-03-29 22:57:55 INFO base.py L1145: start to compute imatrix[0m
2026-03-30T05:58:39.412634+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
chaiml-q235b-judge-dpo-18429-v1-uploader: /usr/local/lib/python3.12/dist-packages/torch/backends/cuda/__init__.py:131: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
chaiml-q235b-judge-dpo-18429-v1-uploader: return torch._C._get_cublas_allow_tf32()
2026-03-30T05:58:43.126809+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/local/lib/python3.12/dist-packages/torch/backends/cuda/__init__.py:131: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
chaiml-q235b-judge-dpo-10887-v1-uploader: return torch._C._get_cublas_allow_tf32()
2026-03-30T05:58:47.417180+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v31
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:53.904000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:53.904000 7 torch/_dynamo/convert_frame.py:1358] [6/8] function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:53.904000 7 torch/_dynamo/convert_frame.py:1358] [6/8] last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1344 # module.imatrix_cnt += input.shape[0] # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:53.904000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:53.904000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:56.905000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:56.905000 7 torch/_dynamo/convert_frame.py:1358] [3/8] function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:56.905000 7 torch/_dynamo/convert_frame.py:1358] [3/8] last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56 # module.imatrix_cnt += input.shape[0] # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:56.905000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-18429-v1-uploader: W0329 22:58:56.905000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:01.199000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:01.199000 7 torch/_dynamo/convert_frame.py:1358] [6/8] function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:01.199000 7 torch/_dynamo/convert_frame.py:1358] [6/8] last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1344 # module.imatrix_cnt += input.shape[0] # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:01.199000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:01.199000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:04.241000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:04.241000 7 torch/_dynamo/convert_frame.py:1358] [3/8] function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:04.241000 7 torch/_dynamo/convert_frame.py:1358] [3/8] last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56 # module.imatrix_cnt += input.shape[0] # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:04.241000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-q235b-judge-dpo-10887-v1-uploader: W0329 22:59:04.241000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-q235b-judge-dpo-18429-v1-uploader: [33;1m2026-03-29 22:59:15 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0[0m
chaiml-q235b-judge-dpo-10887-v1-uploader: [33;1m2026-03-29 22:59:22 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0[0m
2026-03-30T05:59:39.705957+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T05:59:43.424666+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T05:59:47.642554+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v31
2026-03-30T06:00:39.946629+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:00:43.646154+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:00:47.852020+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v31
Inference service chaiml-kimid-v9-opusdv-23365-v31 ready after 220.48678469657898s
Pipeline stage VLLMDeployer completed in 221.50s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.7733423709869385s
Received healthy response to inference request in 3.5882797241210938s
Received healthy response to inference request in 3.4829726219177246s
Received healthy response to inference request in 3.4784739017486572s
Received healthy response to inference request in 1.4965951442718506s
Received healthy response to inference request in 1.571260929107666s
Received healthy response to inference request in 1.4443333148956299s
Received healthy response to inference request in 1.4633769989013672s
Received healthy response to inference request in 1.539154291152954s
Received healthy response to inference request in 1.4922535419464111s
Received healthy response to inference request in 3.6219356060028076s
Received healthy response to inference request in 1.4935519695281982s
Received healthy response to inference request in 1.4801452159881592s
Received healthy response to inference request in 1.5050342082977295s
2026-03-30T06:01:40.184784+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
Received healthy response to inference request in 1.469078779220581s
Received healthy response to inference request in 1.4434051513671875s
2026-03-30T06:01:43.984630+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
Received healthy response to inference request in 1.5442709922790527s
Received healthy response to inference request in 1.53314208984375s
Received healthy response to inference request in 1.446704387664795s
2026-03-30T06:01:48.066702+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v31
Received healthy response to inference request in 1.491999626159668s
Received healthy response to inference request in 1.4635767936706543s
Received healthy response to inference request in 1.55399751663208s
Received healthy response to inference request in 1.566657304763794s
Received healthy response to inference request in 1.4817416667938232s
Received healthy response to inference request in 1.4526476860046387s
Received healthy response to inference request in 1.467857837677002s
Received healthy response to inference request in 1.4424664974212646s
Received healthy response to inference request in 1.5940203666687012s
Received healthy response to inference request in 1.5211901664733887s
Received healthy response to inference request in 1.4484429359436035s
30 requests
0 failed requests
5th percentile: 1.4438228249549865
10th percentile: 1.4464672803878784
20th percentile: 1.4612311363220214
30th percentile: 1.4687124967575074
40th percentile: 1.48789644241333
50th percentile: 1.4950735569000244
60th percentile: 1.5259709358215332
70th percentile: 1.547188949584961
80th percentile: 1.575812816619873
90th percentile: 3.4935033321380615
95th percentile: 3.6067904591560365
99th percentile: 3.7294344091415406
mean time: 1.8450636545817056
Pipeline stage StressChecker completed in 64.42s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.38s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_23365_v31 status is now deployed due to DeploymentManager action
2026-03-30T06:02:40.421372+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:02:44.212598+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:03:40.657016+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:03:44.413355+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:04:40.826385+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:04:44.584252+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:05:41.008420+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:05:44.748972+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:06:41.174464+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:06:44.916051+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:07:41.339420+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:07:45.100240+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:08:41.527189+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:08:45.267454+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:09:41.713105+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:09:45.490490+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:10:41.887298+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:10:45.657971+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:11:42.069927+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:11:45.827281+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:12:42.243537+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:12:46.017250+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:13:42.416449+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:13:46.191526+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:14:42.607733+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:14:46.365309+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:15:42.786513+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:15:46.532376+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:16:42.974732+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:16:46.708353+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:17:43.145811+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:17:46.879057+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:18:43.326364+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:18:47.050489+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:19:43.498735+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:19:47.233229+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:20:43.684203+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:20:47.763351+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:21:43.892560+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:21:47.936152+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:22:44.115975+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:22:48.119552+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:23:44.290550+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:23:48.294104+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:24:44.562484+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:24:48.580790+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:25:44.740550+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:25:48.753520+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:26:44.919167+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:26:48.944373+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:27:45.092067+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:27:49.117353+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:28:45.270513+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:28:49.331891+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:29:45.445043+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:29:49.511211+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:30:45.621336+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:30:49.685653+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:31:45.795301+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:31:49.859304+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:32:45.977509+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:32:50.027002+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:33:46.146280+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:33:50.201419+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:34:46.317557+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:34:50.381308+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:35:46.490544+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:35:50.556755+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:36:46.671004+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:36:50.802036+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:37:46.858005+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:37:50.982224+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:38:47.035344+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:38:51.156513+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:39:47.216264+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:39:51.335008+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:40:47.394577+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:40:51.539120+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:41:47.573966+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:41:51.923418+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:42:47.789459+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:42:52.098985+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:43:48.000821+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:43:52.274149+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:44:48.185757+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:44:52.490759+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:45:48.367658+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:45:52.724840+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:46:48.566872+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:46:52.924710+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:47:48.743999+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:47:53.098198+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:48:48.924591+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:48:53.296122+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:49:49.114972+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:49:53.480699+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:50:49.326748+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:50:53.766694+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:51:49.507749+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:51:53.947887+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:52:49.695984+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:52:54.149146+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:53:49.938308+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:53:54.332871+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:54:50.123227+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:54:54.509770+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:55:50.301634+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:55:54.800537+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:56:50.481989+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:56:54.989735+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:57:50.675479+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:57:55.182304+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:58:50.870649+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:58:55.366128+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T06:59:51.245985+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T06:59:55.695648+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:00:51.478075+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:00:55.904668+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:01:51.698825+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:01:56.116320+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:02:51.919257+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:02:56.334752+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:03:52.134146+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:03:56.543425+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:04:52.337097+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:04:56.758440+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:05:52.554715+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:05:56.993532+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:06:52.774410+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:06:57.222261+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:07:53.073657+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:07:57.461288+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:08:53.299396+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:08:57.676183+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:09:53.510038+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:09:57.887291+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:10:53.721633+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:10:58.108052+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:11:53.940213+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:11:58.318104+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:12:54.158293+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:12:58.530794+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:13:54.388985+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:13:58.757130+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:14:54.704252+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:14:58.983839+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:15:54.921561+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:15:59.224151+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:16:55.145399+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:16:59.465961+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:17:55.354715+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:17:59.681578+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:18:55.679311+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:18:59.927524+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:19:55.924742+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:20:00.147080+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:20:56.250926+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:21:00.547642+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:21:56.497625+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:22:00.773611+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:22:56.717763+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:23:01.005180+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-10887-v1-uploader: Checking if ChaiML/q235b_judge_dpo_lr1-step225-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-10887-v1-uploader: Creating repo ChaiML/q235b_judge_dpo_lr1-step225-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------- 2026-03-30 00:23:34 (0:00:00) ----------
chaiml-q235b-judge-dpo-10887-v1-uploader: Files: hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+27 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-10887-v1-uploader: Workers: hashing: 27 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 98
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------------------------------------------------
2026-03-30T07:23:56.934021+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:24:01.227650+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-18429-v1-uploader: Checking if ChaiML/q235b_judge_dpo_lr1-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-18429-v1-uploader: Creating repo ChaiML/q235b_judge_dpo_lr1-step450-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------- 2026-03-30 00:24:03 (0:00:00) ----------
chaiml-q235b-judge-dpo-18429-v1-uploader: Files: hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+31 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-18429-v1-uploader: Workers: hashing: 27 | get upload mode: 4 | pre-uploading: 1 | committing: 0 | waiting: 94
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-10887-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------- 2026-03-30 00:24:34 (0:01:00) ----------
chaiml-q235b-judge-dpo-10887-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 2/28 (1.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-10887-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------------------------------------------------
2026-03-30T07:24:57.163152+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:25:01.462727+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-18429-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------- 2026-03-30 00:25:03 (0:01:00) ----------
chaiml-q235b-judge-dpo-18429-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 2/28 (1.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-18429-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 100
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-10887-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------- 2026-03-30 00:25:34 (0:02:00) ----------
chaiml-q235b-judge-dpo-10887-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-10887-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------------------------------------------------
2026-03-30T07:25:57.421340+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:26:01.696341+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-18429-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------- 2026-03-30 00:26:03 (0:02:00) ----------
chaiml-q235b-judge-dpo-18429-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-18429-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-10887-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------- 2026-03-30 00:26:34 (0:03:00) ----------
chaiml-q235b-judge-dpo-10887-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-10887-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------------------------------------------------
2026-03-30T07:26:57.664269+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:27:01.923260+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-18429-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------- 2026-03-30 00:27:03 (0:03:00) ----------
chaiml-q235b-judge-dpo-18429-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-18429-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-10887-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------- 2026-03-30 00:27:34 (0:04:00) ----------
chaiml-q235b-judge-dpo-10887-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 20/28 (91.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-10887-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 8 | committing: 0 | waiting: 118
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------------------------------------------------
2026-03-30T07:27:57.908661+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:28:02.168972+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-18429-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------- 2026-03-30 00:28:03 (0:04:00) ----------
chaiml-q235b-judge-dpo-18429-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 26/28 (121.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-18429-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 2 | committing: 0 | waiting: 124
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-10887-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------- 2026-03-30 00:28:34 (0:05:00) ----------
chaiml-q235b-judge-dpo-10887-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-10887-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-q235b-judge-dpo-10887-v1-uploader: ---------------------------------------------------
2026-03-30T07:28:58.154326+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:29:02.409559+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-10887-v1-uploader: Processed model ChaiML/q235b_judge_dpo_lr1-step225-merged in 5675.507s
chaiml-q235b-judge-dpo-18429-v1-uploader:
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------- 2026-03-30 00:29:03 (0:05:00) ----------
chaiml-q235b-judge-dpo-18429-v1-uploader: Files: hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-q235b-judge-dpo-18429-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-q235b-judge-dpo-18429-v1-uploader: ---------------------------------------------------
chaiml-q235b-judge-dpo-10887-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-10887-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-judge-dpo-10887-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-10887-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-10887-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-10887-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-10887-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-judge-dpo-10887-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-judge-dpo-10887-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-judge-dpo-10887-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-judge-dpo-10887-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-judge-dpo-10887-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/special_tokens_map.json
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/quantization_config.json
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/tokenizer_config.json
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/merges.txt
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/chat_template.jinja
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/config.json
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/generation_config.json
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/added_tokens.json
chaiml-q235b-judge-dpo-18429-v1-uploader: Processed model ChaiML/q235b_judge_dpo_lr1-step450-merged in 5703.059s
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/vocab.json
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model.safetensors.index.json
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/tokenizer.json
chaiml-q235b-judge-dpo-18429-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-18429-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-18429-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00027-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-judge-dpo-18429-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-judge-dpo-18429-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-18429-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-18429-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-18429-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-18429-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-18429-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-18429-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-18429-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-18429-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-judge-dpo-18429-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00027-of-00027.safetensors
2026-03-30T07:29:58.471276+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:30:02.680670+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00018-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00001-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00023-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00022-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00012-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00024-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00024-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00022-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-10887-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-10887-v1/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00001-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00012-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00023-of-00027.safetensors
chaiml-q235b-judge-dpo-18429-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-18429-v1/default/model-00018-of-00027.safetensors
Job chaiml-q235b-judge-dpo-10887-v1-uploader completed after 5807.63s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-10887-v1-uploader
Pipeline stage VLLMUploader completed in 5808.67s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.57s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-10887-v1
Waiting for inference service chaiml-q235b-judge-dpo-10887-v1 to be ready
Job chaiml-q235b-judge-dpo-18429-v1-uploader completed after 5815.58s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-18429-v1-uploader
Pipeline stage VLLMUploader completed in 5816.55s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.51s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-18429-v1
Waiting for inference service chaiml-q235b-judge-dpo-18429-v1 to be ready
2026-03-30T07:30:58.732865+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:31:02.937690+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:31:59.060224+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:32:03.197244+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
2026-03-30T07:32:59.356344+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
2026-03-30T07:33:03.493318+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
Inference service chaiml-q235b-judge-dpo-10887-v1 ready after 190.4588463306427s
Pipeline stage VLLMDeployer completed in 191.72s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.152207374572754s
Received healthy response to inference request in 3.4935414791107178s
Received healthy response to inference request in 1.4745607376098633s
Inference service chaiml-q235b-judge-dpo-18429-v1 ready after 200.5065095424652s
Pipeline stage VLLMDeployer completed in 201.68s
run pipeline stage %s
Received healthy response to inference request in 3.6891307830810547s
Running pipeline stage StressChecker
2026-03-30T07:33:59.671974+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
Received healthy response to inference request in 1.4681899547576904s
Received healthy response to inference request in 1.444894552230835s
Received healthy response to inference request in 3.5682661533355713s
Received healthy response to inference request in 1.4453504085540771s
2026-03-30T07:34:03.744767+00:00 monitor updated for chaiml-q235b-judge-dpo-_10887_v1
Received healthy response to inference request in 1.538872480392456s
Received healthy response to inference request in 3.56965970993042s
Received healthy response to inference request in 3.4391579627990723s
Received healthy response to inference request in 1.4501314163208008s
Received healthy response to inference request in 1.4386494159698486s
Received healthy response to inference request in 1.535780668258667s
Received healthy response to inference request in 1.4712133407592773s
Received healthy response to inference request in 1.467632532119751s
Received healthy response to inference request in 1.668400764465332s
Received healthy response to inference request in 3.505707025527954s
Received healthy response to inference request in 1.4728727340698242s
Received healthy response to inference request in 1.4627282619476318s
Received healthy response to inference request in 1.4707348346710205s
Received healthy response to inference request in 2.1072542667388916s
Received healthy response to inference request in 1.5090432167053223s
Received healthy response to inference request in 1.485595703125s
Received healthy response to inference request in 1.4842143058776855s
Received healthy response to inference request in 1.4525480270385742s
Received healthy response to inference request in 1.5017685890197754s
Received healthy response to inference request in 1.625871181488037s
Received healthy response to inference request in 3.6980931758880615s
Received healthy response to inference request in 1.6460351943969727s
Received healthy response to inference request in 1.8529629707336426s
Received healthy response to inference request in 1.6783010959625244s
Received healthy response to inference request in 1.568697214126587s
Received healthy response to inference request in 1.4675183296203613s
Received healthy response to inference request in 1.454408884048462s
Received healthy response to inference request in 1.563420295715332s
Received healthy response to inference request in 1.6171786785125732s
Received healthy response to inference request in 1.5556457042694092s
Received healthy response to inference request in 1.4694926738739014s
Received healthy response to inference request in 1.4576263427734375s
Received healthy response to inference request in 1.5068933963775635s
Received healthy response to inference request in 1.5460865497589111s
Received healthy response to inference request in 1.4725022315979004s
Received healthy response to inference request in 1.4475245475769043s
Received healthy response to inference request in 1.4736566543579102s
Received healthy response to inference request in 1.4602329730987549s
Received healthy response to inference request in 1.4767427444458008s
Received healthy response to inference request in 1.476236343383789s
Received healthy response to inference request in 1.5567409992218018s
Received healthy response to inference request in 1.4733095169067383s
Received healthy response to inference request in 1.474557638168335s
Received healthy response to inference request in 1.5573616027832031s
Received healthy response to inference request in 1.4915335178375244s
Received healthy response to inference request in 1.4786763191223145s
Received healthy response to inference request in 1.5574729442596436s
30 requests
0 failed requests
5th percentile: 1.4475018620491027
10th percentile: 1.453981137275696
20th percentile: 1.4692321300506592
Received healthy response to inference request in 1.5116493701934814s
30th percentile: 1.472761583328247
40th percentile: 1.474559497833252
50th percentile: 1.487873911857605
60th percentile: 1.5197381973266602
70th percentile: 1.5608402252197264
80th percentile: 1.7053132057189946
90th percentile: 3.5816068172454836
95th percentile: 3.6940600991249086
Received healthy response to inference request in 1.6011390686035156s
99th percentile: 4.020514256954193
mean time: 1.8822426398595173
Pipeline stage StressChecker completed in 69.89s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 1.6384694576263428s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.51s
Shutdown handler de-registered
chaiml-q235b-judge-dpo-_10887_v1 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 1.4634733200073242s
Received healthy response to inference request in 1.493271827697754s
30 requests
0 failed requests
5th percentile: 1.4497851133346558
10th percentile: 1.4571185111999512
20th percentile: 1.4633243083953857
30th percentile: 1.4726806640625
2026-03-30T07:34:59.921941+00:00 monitor updated for chaiml-q235b-judge-dpo-_18429_v1
40th percentile: 1.4828279495239258
50th percentile: 1.5067089796066284
60th percentile: 1.5499102115631103
70th percentile: 1.574735927581787
80th percentile: 1.6399826049804687
90th percentile: 2.2404446363449115
95th percentile: 3.475759947299957
99th percentile: 3.5501240062713624
mean time: 1.7371203422546386
Pipeline stage StressChecker completed in 63.19s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.04s
Shutdown handler de-registered
chaiml-q235b-judge-dpo-_18429_v1 status is now deployed due to DeploymentManager action
chaiml-q235b-judge-dpo-_18429_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-q235b-judge-dpo-_18429_v1 status is now torndown due to DeploymentManager action