chaiml-q235b-judge-dpo-_74524

submission_id: chaiml-q235b-judge-dpo-_74524_v2
developer_uid: rirv938
status: torndown
model_repo: ChaiML/q235b_judge_dpo-step450-merged
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</think>', '</s>', '<|user|>', '<|im_end|>', '####', '<|assistant|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
timestamp: 2026-03-29T17:50:56+00:00
model_name: chaiml-q235b-judge-dpo-_74524_v2
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-judge-dpo-74524-v2-uploader
Waiting for job on chaiml-q235b-judge-dpo-74524-v2-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v29-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v29-uploader to finish
chaiml-q235b-judge-dpo-47447-v2-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-74524-v2-uploader: Using quantization_mode: w4a16
chaiml-q235b-judge-dpo-74524-v2-uploader: Checking if ChaiML/q235b_judge_dpo-step450-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-74524-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-q235b-judge-dpo-74524-v2-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo-step450-merged-W4A16...
chaiml-kimid-v9-opusdv-23365-v29-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v29-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v9-opusdv-23365-v29-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v29-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
chaiml-q235b-judge-dpo-47447-v2-uploader: Checking if ChaiML/q235b_judge_dpo-step225-merged-W4A16 already exists in ChaiML
chaiml-q235b-judge-dpo-47447-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-q235b-judge-dpo-47447-v2-uploader: Downloading snapshot of ChaiML/q235b_judge_dpo-step225-merged-W4A16...
2026-03-29T17:06:21.466672+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:06:25.474641+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:06:29.676418+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
chaiml-q235b-judge-dpo-47447-v2-uploader: Downloaded in 52.915s
chaiml-q235b-judge-dpo-47447-v2-uploader: Processed model ChaiML/q235b_judge_dpo-step225-merged in 53.643s
chaiml-q235b-judge-dpo-47447-v2-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-47447-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-judge-dpo-47447-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-judge-dpo-47447-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-judge-dpo-47447-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-47447-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-47447-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-47447-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-47447-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-47447-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-judge-dpo-47447-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-judge-dpo-47447-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-judge-dpo-47447-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-judge-dpo-47447-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-judge-dpo-47447-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/.gitattributes
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/generation_config.json
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/special_tokens_map.json
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/quantization_config.json
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/chat_template.jinja
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/config.json
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/added_tokens.json
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/tokenizer_config.json
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/vocab.json
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/merges.txt
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/tokenizer.json
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model.safetensors.index.json
chaiml-q235b-judge-dpo-74524-v2-uploader: Downloaded in 53.881s
chaiml-q235b-judge-dpo-74524-v2-uploader: Processed model ChaiML/q235b_judge_dpo-step450-merged in 54.474s
chaiml-q235b-judge-dpo-74524-v2-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-74524-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-74524-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-judge-dpo-74524-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-judge-dpo-74524-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-judge-dpo-74524-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-74524-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-74524-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-74524-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-judge-dpo-74524-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-74524-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-74524-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-judge-dpo-74524-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-judge-dpo-74524-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-judge-dpo-74524-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-judge-dpo-74524-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-judge-dpo-74524-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-judge-dpo-74524-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-judge-dpo-74524-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/added_tokens.json
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/.gitattributes
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/config.json
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/generation_config.json
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/special_tokens_map.json
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/chat_template.jinja
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/merges.txt
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model.safetensors.index.json
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/tokenizer_config.json
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/vocab.json
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/quantization_config.json
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/tokenizer.json
2026-03-29T17:07:21.638773+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:07:25.642870+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00023-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00018-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00012-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00024-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00001-of-00027.safetensors
2026-03-29T17:07:29.859397+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00022-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-47447-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00002-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00022-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00008-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00006-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00018-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00004-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00023-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00014-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00010-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00009-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00007-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00012-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00001-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00011-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00003-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00020-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00021-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00017-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00019-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00005-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00016-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00026-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00015-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00025-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00013-of-00027.safetensors
chaiml-q235b-judge-dpo-74524-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default/model-00024-of-00027.safetensors
Job chaiml-q235b-judge-dpo-47447-v2-uploader completed after 140.64s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-47447-v2-uploader
Pipeline stage VLLMUploader completed in 141.47s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.95s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-47447-v2
Waiting for inference service chaiml-q235b-judge-dpo-47447-v2 to be ready
Job chaiml-q235b-judge-dpo-74524-v2-uploader completed after 142.78s with status: succeeded
Stopping job with name chaiml-q235b-judge-dpo-74524-v2-uploader
Pipeline stage VLLMUploader completed in 143.66s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.46s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-judge-dpo-74524-v2
Waiting for inference service chaiml-q235b-judge-dpo-74524-v2 to be ready
2026-03-29T17:08:21.839802+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:08:25.837029+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:08:30.068295+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
chaiml-kimid-v9-opusdv-23365-v29-uploader: Downloaded in 142.389s
chaiml-kimid-v9-opusdv-23365-v29-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 142.931s
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/added_tokens.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/chat_template.jinja
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/config.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/generation_config.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/special_tokens_map.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/merges.txt
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/vocab.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/tokenizer_config.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/quantization_config.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model.safetensors.index.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/tokenizer.json
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/.gitattributes
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00006-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v29-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v29/default/model-00011-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-23365-v29-uploader completed after 231.18s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v29-uploader
2026-03-29T17:09:22.091076+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
Pipeline stage VLLMUploader completed in 232.24s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.65s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v29
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v29 to be ready
2026-03-29T17:09:26.100297+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:09:30.301997+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:10:22.444436+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:10:26.347640+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:10:30.630830+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:11:22.741214+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:11:37.192382+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:11:40.262957+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:12:22.987692+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:12:37.479698+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:12:40.504555+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:13:23.237985+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:13:38.011614+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:13:40.777131+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:14:23.487862+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:14:38.274390+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:14:41.020618+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:15:23.737156+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:15:38.568999+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:15:41.278893+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:16:24.029384+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:16:38.820868+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:16:41.529357+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:17:24.350892+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:17:39.075914+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:17:41.779547+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:18:24.661594+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:18:39.345414+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:18:42.069081+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:19:24.984399+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:19:39.638579+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:19:42.322302+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:20:25.240964+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:20:39.892220+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:20:42.596324+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:21:25.549463+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:21:40.185979+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:21:42.853655+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:22:25.851519+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:22:40.462711+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:22:43.118131+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:23:26.201825+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:23:40.719407+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:23:43.384153+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:24:26.506653+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:24:41.031626+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:24:43.651035+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:25:26.782755+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:25:41.537792+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:25:45.208256+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:26:29.877316+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:26:41.798509+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:26:45.470136+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:27:30.140940+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:27:42.059077+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:27:45.738730+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:28:30.407014+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:28:42.320799+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:28:45.999978+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:29:30.662123+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:29:42.630014+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:29:46.272535+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:30:30.945344+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:30:42.892505+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:30:46.529573+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:31:31.250665+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:31:43.156499+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:31:46.826433+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:32:31.522799+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:32:43.425734+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:32:47.092183+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:33:31.836334+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:33:43.688972+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:33:47.359263+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:34:32.132441+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:34:43.941962+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:34:47.616596+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:35:32.403363+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:35:44.580732+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:35:47.881444+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:36:32.717515+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:36:44.861845+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:36:48.158550+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:37:32.987307+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:37:45.129437+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:37:48.418033+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:38:33.261445+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:38:45.457187+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:38:48.713619+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:39:33.536859+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:39:45.772883+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:39:48.992054+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:40:33.832543+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:40:46.075372+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:40:49.271520+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:41:45.899378+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:41:46.345402+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:41:49.981660+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:42:46.168008+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:42:46.620796+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:42:50.280514+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:43:46.475249+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:43:46.889246+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:43:50.586649+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:44:46.811186+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:44:47.187436+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:44:50.877435+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:45:47.095295+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:45:47.832632+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:45:51.162193+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:46:47.440571+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:46:48.101482+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
2026-03-29T17:46:51.440721+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
2026-03-29T17:47:47.746846+00:00 monitor updated for chaiml-q235b-judge-dpo-_47447_v2
2026-03-29T17:47:48.364861+00:00 monitor updated for chaiml-q235b-judge-dpo-_74524_v2
Tearing down inference service chaiml-q235b-judge-dpo-47447-v2
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-q235b-judge-dpo-47447-v2.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'40m\'}, \'creationTimestamp\': \'2026-03-29T17:07:47Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'istio.io/rev\': \'prod-canary\', \'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:istio.io/rev\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-03-29T17:07:47Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-03-29T17:07:47Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-03-29T17:19:54Z\'}], \'name\': \'chaiml-q235b-judge-dpo-47447-v2\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'1301556539\', \'uid\': \'e576b63d-c067-4c66-b279-2d3985847d37\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'8\', \'--gpu-memory-utilization\', \'0.9\', \'--max-model-len\', \'10240\', \'--max-num-batched-tokens\', \'10240\', \'--max-num-seqs\', \'64\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/q235b_judge_dpo-step225-merged\', \'--model-loader-extra-config\', \'{"distributed": true, "concurrency": 2}\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'8\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'1\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'CWZAGMHZXKZRFGJK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'cwoAeWzp46q4O0sTNXOEuZ1MvZzKEFlS9DtEhnTldKp\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://cwobject.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.17.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}, \'requests\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 5, \'minReplicas\': 0, \'priorityClassName\': \'chaiverse\', \'timeout\': 20, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'163Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-q235b-judge-dpo-47447-v2-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'message\': \'Revision "chaiml-q235b-judge-dpo-47447-v2-predictor-00001" failed with message: 0/276 nodes are available: 1 Insufficient cpu, 122 Insufficient nvidia.com/gpu, 147 node(s) didn\\\'t match Pod\\\'s node affinity/selector, 2 node(s) had untolerated taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, 5 node(s) were unschedulable. preemption: 0/276 nodes are available: 122 Insufficient nvidia.com/gpu, 154 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-47447-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-47447-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-47447-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
2026-03-29T17:47:51.703253+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v29
Pipeline stage VLLMDeleter completed in 0.59s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/.gitattributes from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/added_tokens.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/merges.txt from bucket guanaco-vllm-models
Tearing down inference service chaiml-q235b-judge-dpo-74524-v2
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00001-of-00027.safetensors from bucket guanaco-vllm-models
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-q235b-judge-dpo-74524-v2.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'40m\'}, \'creationTimestamp\': \'2026-03-29T17:07:51Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'istio.io/rev\': \'prod-canary\', \'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:istio.io/rev\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-03-29T17:07:51Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-03-29T17:07:51Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-03-29T17:47:06Z\'}], \'name\': \'chaiml-q235b-judge-dpo-74524-v2\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'1301610364\', \'uid\': \'a9b2bafa-3313-4295-89f0-bad455da69c9\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-74524-v2/default\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'8\', \'--gpu-memory-utilization\', \'0.9\', \'--max-model-len\', \'10240\', \'--max-num-batched-tokens\', \'10240\', \'--max-num-seqs\', \'64\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/q235b_judge_dpo-step450-merged\', \'--model-loader-extra-config\', \'{"distributed": true, "concurrency": 2}\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'8\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'1\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'CWZAGMHZXKZRFGJK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'cwoAeWzp46q4O0sTNXOEuZ1MvZzKEFlS9DtEhnTldKp\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://cwobject.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.17.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}, \'requests\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 5, \'minReplicas\': 0, \'priorityClassName\': \'chaiverse\', \'timeout\': 20, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'163Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-q235b-judge-dpo-74524-v2-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-03-29T17:47:06Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:47:06Z\', \'message\': \'Revision "chaiml-q235b-judge-dpo-74524-v2-predictor-00001" failed with message: 0/276 nodes are available: 1 Insufficient cpu, 122 Insufficient nvidia.com/gpu, 147 node(s) didn\\\'t match Pod\\\'s node affinity/selector, 2 node(s) had untolerated taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, 5 node(s) were unschedulable. preemption: 0/276 nodes are available: 122 Insufficient nvidia.com/gpu, 154 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:47:06Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-74524-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:47:06Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-74524-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:47:06Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-74524-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-03-29T17:47:06Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
run pipeline stage %s
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00002-of-00027.safetensors from bucket guanaco-vllm-models
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00003-of-00027.safetensors from bucket guanaco-vllm-models
Pipeline stage VLLMDeleter completed in 0.65s
run pipeline stage %s
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00004-of-00027.safetensors from bucket guanaco-vllm-models
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00005-of-00027.safetensors from bucket guanaco-vllm-models
Cleaning model data from model cache
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/.gitattributes from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00006-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/added_tokens.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00007-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00008-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/merges.txt from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00001-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00009-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00002-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00010-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00003-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00011-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00004-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00012-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00005-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00013-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00006-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00014-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00007-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00015-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00008-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00016-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00009-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00017-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00010-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00018-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00011-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00019-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00012-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00020-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00013-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00021-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00014-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00022-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00015-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00023-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00016-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00024-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00017-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00025-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00018-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00026-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00019-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model-00027-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00020-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/quantization_config.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00021-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/tokenizer_config.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00022-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-47447-v2/default/vocab.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 23.54s
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00023-of-00027.safetensors from bucket guanaco-vllm-models
Shutdown handler de-registered
DeploymentError('Timeout to start the InferenceService chaiml-q235b-judge-dpo-47447-v2.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'40m\'}, \'creationTimestamp\': \'2026-03-29T17:07:47Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'istio.io/rev\': \'prod-canary\', \'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:istio.io/rev\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-03-29T17:07:47Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-03-29T17:07:47Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-03-29T17:19:54Z\'}], \'name\': \'chaiml-q235b-judge-dpo-47447-v2\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'1301556539\', \'uid\': \'e576b63d-c067-4c66-b279-2d3985847d37\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-q235b-judge-dpo-47447-v2/default\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'8\', \'--gpu-memory-utilization\', \'0.9\', \'--max-model-len\', \'10240\', \'--max-num-batched-tokens\', \'10240\', \'--max-num-seqs\', \'64\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/q235b_judge_dpo-step225-merged\', \'--model-loader-extra-config\', \'{"distributed": true, "concurrency": 2}\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'8\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'1\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'CWZAGMHZXKZRFGJK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'cwoAeWzp46q4O0sTNXOEuZ1MvZzKEFlS9DtEhnTldKp\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'https://cwobject.com\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.17.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}, \'requests\': {\'cpu\': \'16\', \'memory\': \'163Gi\', \'nvidia.com/gpu\': \'8\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 5, \'minReplicas\': 0, \'priorityClassName\': \'chaiverse\', \'timeout\': 20, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'163Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-q235b-judge-dpo-47447-v2-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'message\': \'Revision "chaiml-q235b-judge-dpo-47447-v2-predictor-00001" failed with message: 0/276 nodes are available: 1 Insufficient cpu, 122 Insufficient nvidia.com/gpu, 147 node(s) didn\\\'t match Pod\\\'s node affinity/selector, 2 node(s) had untolerated taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, 5 node(s) were unschedulable. preemption: 0/276 nodes are available: 122 Insufficient nvidia.com/gpu, 154 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-47447-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-47447-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'message\': \'Configuration "chaiml-q235b-judge-dpo-47447-v2-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-03-29T17:19:54Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00024-of-00027.safetensors from bucket guanaco-vllm-models
chaiml-q235b-judge-dpo-_47447_v2 status is now failed due to DeploymentManager action
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00025-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00026-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model-00027-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/quantization_config.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/tokenizer_config.json from bucket guanaco-vllm-models
Deleting key chaiml-q235b-judge-dpo-74524-v2/default/vocab.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 23.43s
Shutdown handler de-registered
chaiml-q235b-judge-dpo-_74524_v2 status is now failed due to DeploymentManager action
chaiml-q235b-judge-dpo-_74524_v2 status is now torndown due to DeploymentManager action