submission_id: chaiml-pony-v2-g46-lr1e_80834_v1
developer_uid: chai_backend_admin
status: torndown
model_repo: ChaiML/pony-v2-g46-lr1e4ep1r64b16
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_end|>', 'You:', '<|im_start|>', '###'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '[gMASK]<sop><|system|>\nRespond as a high quality storyteller.<|user|>\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': False}
timestamp: 2026-03-02T18:46:01+00:00
model_name: chaiml-pony-v2-g46-lr1e_80834_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-g46-lr1e-80834-v1-uploader
Waiting for job on chaiml-pony-v2-g46-lr1e-80834-v1-uploader to finish
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Using quantization_mode: fp8
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Checking if ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Downloading snapshot of ChaiML/pony-v2-g46-lr1e4ep1r64b16...
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Downloaded in 261.400s
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Loading /tmp/model_input...
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Applying quantization...
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: 2026-03-02T09:31:03.055483-0800 | reset | INFO - Compression lifecycle reset
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: 2026-03-02T09:31:03.121995-0800 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: 2026-03-02T09:31:11.697578-0800 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: 2026-03-02T09:31:17.846219-0800 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Saving to /dev/shm/model_output...
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: 2026-03-02T09:31:17.877639-0800 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: 2026-03-02T09:31:59.453273-0800 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: 2026-03-02T09:31:59.453759-0800 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: 2026-03-02T09:43:13.224725-0800 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: 2026-03-02T09:43:19.851156-0800 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Saving to /dev/shm/model_output...
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: 2026-03-02T09:43:19.877714-0800 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Pushing to ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Checking if ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Creating repo ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8 and uploading /dev/shm/model_output to it
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:12:56 (0:00:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: 







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:13:56 (0:01:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 2/73 (1.6G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 71 | committing: 0 | waiting: 55
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:14:56 (0:02:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 10/73 (41.5G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 63 | committing: 0 | waiting: 63
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:15:56 (0:03:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 15/73 (66.5G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 58 | committing: 0 | waiting: 68
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:16:56 (0:04:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 18/73 (81.5G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 55 | committing: 0 | waiting: 71
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:17:56 (0:05:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 26/73 (121.5G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 47 | committing: 0 | waiting: 79
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:18:56 (0:06:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 34/73 (161.4G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 39 | committing: 0 | waiting: 87
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:19:56 (0:07:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 42/73 (201.4G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 31 | committing: 0 | waiting: 95
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:20:56 (0:08:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 46/73 (221.3G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 27 | committing: 0 | waiting: 99
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:21:56 (0:09:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 51/73 (246.3G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 22 | committing: 0 | waiting: 104
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:22:56 (0:10:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 58/73 (281.3G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 15 | committing: 0 | waiting: 111
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:23:56 (0:11:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 66/73 (321.2G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 7 | committing: 0 | waiting: 119
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:24:56 (0:12:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Pushing to ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Checking if ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Creating repo ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 and uploading /dev/shm/model_output to it
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:25:22 (0:00:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------- 2026-03-02 10:25:56 (0:13:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (206.4G/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: 







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:26:22 (0:01:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:                              Processed model ChaiML/opusd-v4-g46-lr1e4ep1r64b16 in 4988.399s
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:27:22 (0:02:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-opusd-v4-g46-lr1-69112-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/chat_template.jinja
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/config.json
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/tokenizer_config.json
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/recipe.yaml
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/generation_config.json
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model.safetensors.index.json
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/special_tokens_map.json
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/tokenizer.json
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00072-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00071-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:28:22 (0:03:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 14/73 (60.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 59 | committing: 0 | waiting: 67
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00009-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00062-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00055-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00044-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00011-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00028-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00061-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00003-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00050-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00054-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00005-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00033-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00019-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00056-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00008-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00068-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00048-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00066-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00007-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00045-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00049-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00027-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00006-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00036-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00059-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00047-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00053-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00060-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00065-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00031-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00001-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00038-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00024-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00051-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00046-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00026-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00039-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00032-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00022-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00014-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00069-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00063-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00042-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00017-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00070-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00010-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00040-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00018-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00035-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00013-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00037-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00023-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00015-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00043-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00041-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:29:22 (0:04:00) ----------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00030-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 19/73 (85.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00057-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 54 | committing: 0 | waiting: 72
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00025-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00004-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00034-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00016-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00012-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00052-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00067-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00058-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00064-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00020-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00021-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00002-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v4-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00029-of-00072.safetensors
Job chaiml-opusd-v4-g46-lr1-69112-v4-uploader completed after 5214.4s with status: succeeded
Stopping job with name chaiml-opusd-v4-g46-lr1-69112-v4-uploader
Pipeline stage VLLMUploader completed in 5216.67s
run pipeline stage %s
Running pipeline stage VLLMTemplater
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=KubeFormatError("kube_kwargs keys do not match the kube template, expected: {'engine_args', 'model_repo', 'aws_secret_access_key', 'cpus', 'num_gpus', 'aws_access_key_id', 'endpoint', 'aws_endpoint_url', 'additional_isvc_labels', 'download_to_local', 'gpu_type', 'autoscaling_hard_target', 'initial_scale', 'target_latency', 'preferred_gpu_type', 'memory', 'max_replicas', 'progress_deadline', 'priority_class_name', 'min_replicas', 'docker_image'}, given: {'engine_args', 'model_repo', 'aws_secret_access_key', 'cpus', 'gpu_manufacturer', 'num_gpus', 'aws_access_key_id', 'endpoint', 'aws_endpoint_url', 'additional_isvc_labels', 'download_to_local', 'gpu_type', 'autoscaling_hard_target', 'initial_scale', 'target_latency', 'preferred_gpu_type', 'memory', 'max_replicas', 'progress_deadline', 'priority_class_name', 'min_replicas', 'docker_image'}")
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 1.43s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00001-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00002-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00003-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00004-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00005-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00006-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00007-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00008-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00009-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00010-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00011-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00012-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00013-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00014-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00015-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00016-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00017-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00018-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00019-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00020-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:30:22 (0:05:00) ----------
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00021-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (124.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00022-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00023-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00024-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00025-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00026-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00027-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00028-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00029-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00030-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00031-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00032-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00033-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00034-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00035-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00036-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00037-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00038-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00039-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00040-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00041-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00042-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00043-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00044-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00045-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00046-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00047-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00048-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00049-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00050-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00051-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00052-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00053-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00054-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00055-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00056-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00057-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00058-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00059-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00060-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00061-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00062-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00063-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00064-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00065-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00066-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:31:22 (0:06:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00067-of-00072.safetensors from bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00068-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00069-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00070-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00071-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model-00072-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/recipe.yaml from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-opusd-v4-g46-lr1-69112-v4/default/tokenizer_config.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 102.47s
Shutdown handler de-registered
KubeFormatError("kube_kwargs keys do not match the kube template, expected: {'engine_args', 'model_repo', 'aws_secret_access_key', 'cpus', 'num_gpus', 'aws_access_key_id', 'endpoint', 'aws_endpoint_url', 'additional_isvc_labels', 'download_to_local', 'gpu_type', 'autoscaling_hard_target', 'initial_scale', 'target_latency', 'preferred_gpu_type', 'memory', 'max_replicas', 'progress_deadline', 'priority_class_name', 'min_replicas', 'docker_image'}, given: {'engine_args', 'model_repo', 'aws_secret_access_key', 'cpus', 'gpu_manufacturer', 'num_gpus', 'aws_access_key_id', 'endpoint', 'aws_endpoint_url', 'additional_isvc_labels', 'download_to_local', 'gpu_type', 'autoscaling_hard_target', 'initial_scale', 'target_latency', 'preferred_gpu_type', 'memory', 'max_replicas', 'progress_deadline', 'priority_class_name', 'min_replicas', 'docker_image'}")
chaiml-opusd-v4-g46-lr1_69112_v4 status is now failed due to DeploymentManager action
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:32:22 (0:07:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 42/73 (199.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 31 | committing: 0 | waiting: 95
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:33:22 (0:08:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:34:22 (0:09:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 51/73 (244.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 22 | committing: 0 | waiting: 104
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:35:22 (0:10:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 59/73 (284.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 14 | committing: 0 | waiting: 112
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:36:22 (0:11:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 67/73 (324.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 6 | committing: 0 | waiting: 120
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:37:22 (0:12:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              







chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------- 2026-03-02 10:38:22 (0:13:00) ----------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.8G/354.7G) | ignored: 0
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: ---------------------------------------------------
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:                              Processed model ChaiML/pony-v2-g46-lr1e4ep1r64b16 in 4415.366s
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-g46-lr1e-80834-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/config.json
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/special_tokens_map.json
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model.safetensors.index.json
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/chat_template.jinja
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/generation_config.json
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/tokenizer_config.json
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/recipe.yaml
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/tokenizer.json
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00072-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00071-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00027-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00008-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00004-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00068-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00022-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00037-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00039-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00053-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00051-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00040-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00049-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00057-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00045-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00029-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00030-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00066-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00069-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00064-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00032-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00012-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00028-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00060-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00059-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00054-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00001-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00034-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00047-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00061-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00052-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00009-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00041-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00055-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00067-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00002-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00036-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00018-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00023-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00003-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00070-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00021-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00015-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00005-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00017-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00048-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00013-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00062-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00065-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00019-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00042-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00016-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00063-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00043-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00031-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00025-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00011-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00026-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00056-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00010-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00044-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00038-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00035-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00033-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00046-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00020-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00006-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00007-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00050-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00058-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00024-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00014-of-00072.safetensors
Job chaiml-pony-v2-g46-lr1e-80834-v1-uploader completed after 4621.9s with status: succeeded
Stopping job with name chaiml-pony-v2-g46-lr1e-80834-v1-uploader
Pipeline stage VLLMUploader completed in 4623.88s
run pipeline stage %s
Running pipeline stage VLLMTemplater
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=KubeFormatError("kube_kwargs keys do not match the kube template, expected: {'engine_args', 'model_repo', 'aws_secret_access_key', 'cpus', 'num_gpus', 'aws_access_key_id', 'endpoint', 'aws_endpoint_url', 'additional_isvc_labels', 'download_to_local', 'gpu_type', 'autoscaling_hard_target', 'initial_scale', 'target_latency', 'preferred_gpu_type', 'memory', 'max_replicas', 'progress_deadline', 'priority_class_name', 'min_replicas', 'docker_image'}, given: {'engine_args', 'model_repo', 'aws_secret_access_key', 'cpus', 'gpu_manufacturer', 'num_gpus', 'aws_access_key_id', 'endpoint', 'aws_endpoint_url', 'additional_isvc_labels', 'download_to_local', 'gpu_type', 'autoscaling_hard_target', 'initial_scale', 'target_latency', 'preferred_gpu_type', 'memory', 'max_replicas', 'progress_deadline', 'priority_class_name', 'min_replicas', 'docker_image'}")
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.86s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00001-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00002-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00003-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00004-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00005-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00006-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00007-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00008-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00009-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00010-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00011-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00012-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00013-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00014-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00015-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00016-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00017-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00018-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00019-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00020-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00021-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00022-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00023-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00024-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00025-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00026-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00027-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00028-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00029-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00030-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00031-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00032-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00033-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00034-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00035-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00036-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00037-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00038-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00039-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00040-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00041-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00042-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00043-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00044-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00045-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00046-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00047-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00048-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00049-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00050-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00051-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00052-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00053-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00054-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00055-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00056-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00057-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00058-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00059-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00060-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00061-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00062-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00063-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00064-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00065-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00066-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00067-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00068-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00069-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00070-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00071-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model-00072-of-00072.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/recipe.yaml from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-pony-v2-g46-lr1e-80834-v1/default/tokenizer_config.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 90.66s
Shutdown handler de-registered
chaiml-pony-v2-g46-lr1e_80834_v1 status is now failed due to DeploymentManager action
chaiml-pony-v2-g46-lr1e_80834_v1 status is now torndown due to DeploymentManager action