Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-llama31-mer-v2-44570-v78-uploader
Waiting for job on chaiml-llama31-mer-v2-44570-v78-uploader to finish
chaiml-llama31-mer-v2-44570-v78-uploader: Using quantization_mode: none
chaiml-llama31-mer-v2-44570-v78-uploader: Downloading snapshot of ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572...
chaiml-llama31-mer-v2-44570-v78-uploader: Downloaded in 6.132s
chaiml-llama31-mer-v2-44570-v78-uploader: Processed model ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572 in 11.666s
chaiml-llama31-mer-v2-44570-v78-uploader: creating bucket guanaco-vllm-models
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-llama31-mer-v2-44570-v78-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-llama31-mer-v2-44570-v78-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-llama31-mer-v2-44570-v78-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-llama31-mer-v2-44570-v78-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-llama31-mer-v2-44570-v78-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: 890e6faa-0249-9550-a255-c0121ad4c050, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 730199c4-782f-9132-b59e-eee37abc3801, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 54547842-c588-935c-8009-84022450d213, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: a77cd78a-1976-972d-95fc-bfeed3729a5e, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: cfea3fe9-a67a-9f27-a572-0f47cbb79064, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: d8a32565-e470-9721-82dc-207432ee575d, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: c7faa393-64c4-9f62-bbb4-200717115902, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 096c656d-db6c-9ee1-8017-1318ba189f43, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 142233fa-bd0c-97e7-862c-bf864728d53f, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 46d05c98-4080-9d9c-97e7-f94960d60ef8, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 6b4486a1-3664-9e7c-ac89-5386223e6caf, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: b9f6b797-cd90-9051-ab7f-bf5647d89dd7, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: aa36e1ff-0e9b-95ba-808a-1be72141dcfd, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 435ae722-c35d-9488-9413-f851a497587e, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 2d1b2c24-0dff-97ec-a1e2-ee54e8b7d7ad, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 7c99f626-2677-9d17-abee-e609c07b3a88, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 2a2a9f59-c3b8-926a-99c8-84fc4bbf56c2, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 808a786b-2aea-9821-8c88-7c8d22eea31f, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: a7bae8e4-dbbe-9c46-b249-9b4843c6dc32, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: a4d66945-ea8a-944a-a384-481a6cf73d01, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 44d667c5-40d9-9ffe-a836-1ac5cbaba266, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: e929342b-5d3c-93e5-a2d8-79995074a703, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 09eaecc8-a0a5-913b-a74c-10abf3b45e1e, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: a42952cf-15b0-980e-8bf4-439082180906, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 3/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: 9c1200e0-5b27-9e68-a300-cf86e1d8369b, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 80b2cbce-adfd-9b3e-8021-0a8adc35e46b, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: c0d7b02d-89f7-95f8-a348-b6ea8e066899, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: cc858560-7130-9daa-a7b5-1077519a3ff1, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 1f7c2ca2-525d-93d5-ab80-38ed7eea5f1f, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 0898f2dc-e9ea-96be-86d4-de70c6576742, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 4ac91670-c3d3-993f-80ea-cb647d6d6744, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 7ad45e57-5dc8-953f-9c30-eaa9609fe236, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 102d4633-bb64-92e4-806c-a410d4ce450b, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 2e1cd6c5-7f3c-954d-8d28-41cc8e335348, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 7b234802-e961-9126-bd13-deb8ff4d5bbd, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: f8ba7269-cccb-9002-add2-736f7d02e595, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 4/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: cfd8ac5c-369f-94cf-98e8-7c32c7991021, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: f436354f-5fff-9a73-b7ce-318799912bcd, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 758884c1-617b-9f17-b3d4-addc7835c408, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 00f172f4-06f1-99dd-af60-201543b1386f, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 8651d74b-4066-961e-b76b-1239fa64724e, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 2b1ed152-dea8-94f1-a5ed-f8ddf99688ea, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 69fc0862-fdaa-90e5-86b7-c80abcc5c98e, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 4cf4b8b1-ab67-93fb-9ca6-72a68f336b92, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: ea8f256e-49e1-9574-8626-e19871afc369, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 2a11a105-cc64-9bb5-8394-801f4754a032, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: f197e398-2828-9aff-bf6d-cb22b2ba75b6, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: e182c037-6cf5-9d21-a883-000b848fb9b7, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 5/5 exited 1, no more retries left.
Job chaiml-llama31-mer-v2-44570-v78-uploader completed after 52.14s with status: failed
Job failed chaiml-llama31-mer-v2-44570-v78-uploader:
Stopping job with name chaiml-llama31-mer-v2-44570-v78-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-llama31-mer-v2-44570-v78-uploader
Waiting for job on chaiml-llama31-mer-v2-44570-v78-uploader to finish
chaiml-llama31-mer-v2-44570-v78-uploader: Using quantization_mode: none
chaiml-llama31-mer-v2-44570-v78-uploader: Downloading snapshot of ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572...
chaiml-llama31-mer-v2-44570-v78-uploader: Downloaded in 5.732s
chaiml-llama31-mer-v2-44570-v78-uploader: Processed model ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572 in 11.444s
chaiml-llama31-mer-v2-44570-v78-uploader: creating bucket guanaco-vllm-models
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-llama31-mer-v2-44570-v78-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-llama31-mer-v2-44570-v78-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-llama31-mer-v2-44570-v78-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-llama31-mer-v2-44570-v78-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-llama31-mer-v2-44570-v78-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: 82190138-3daf-9d78-8503-351b71ade043, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 56de78aa-2227-980f-80df-b315dbbf630e, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 707d1ef4-3194-9398-90a7-c53ecc861d21, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 53b71ed5-ad85-96bd-b4fe-d3efc5577d45, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 132cee9a-7b70-9257-9cfe-d216cf3090b6, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: bb3a3553-ba81-9d17-93ed-ca7b10af73ac, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: a0f7a109-e8ee-9882-a51d-486727e2ecfd, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: e037a270-f314-9a22-8b24-e479c2ef005e, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 315e854f-1f50-9aea-88f4-9ba131073d8a, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: ca13c890-1f17-980d-bd0e-1c0b4bb91b75, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 65b4f0d6-d273-900a-bf9d-002d4f154139, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: bad5e969-624d-9851-a4a1-3c12a6f69495, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: 064c3039-2aee-9e0f-aa06-2c1fb799de7d, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: e625b2de-f8e3-9932-838c-49ba90a7a4a4, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 81a39b79-2d96-93a1-b0c2-bdda9fd7a98c, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: c1a9f4a3-3479-9a49-847d-1f4b786de475, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: cd00aea0-cd7d-9fae-b8b1-2387c45bad79, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: e96cae95-90c8-9bee-b8f8-7be67abe8806, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 6b454715-abda-985e-8260-771edd98f12c, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 613339d7-9bed-9ddb-bcbd-ecfbf054ffc8, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: eef0f35a-04cd-931c-a032-6603250e2bf2, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 4a62c259-62ea-9163-b26b-ec318f999149, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: c8d71ee1-390f-9387-b71e-0b867eda8f48, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: b319075f-6701-9ff5-ad40-c7344f9d9fe3, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 4/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: 26ba7910-7bde-98ee-98c5-6351ef73548c, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: aaf79820-a69a-97d0-971b-db7589b90b65, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: de357814-4ca1-94cb-a547-2a1f8736a90c, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 5e267439-4aa4-9870-8e9a-7bdb21e02342, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 92e66620-b088-928b-9149-bcbd6c33aa25, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: a0700eef-b419-9d20-ba4a-2b415f941162, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 00fbd8b1-4f18-99c9-9417-9e05546ee176, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 4b148624-bf79-9d61-ba44-045c8fceffa2, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 2306d7fb-81b4-9b0d-95c8-1f7d70789f03, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 816240a9-1b93-9e2e-88f9-0db79c862ccc, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 307287ff-6481-9b37-88f1-89503cecab59, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: b22699ef-8145-943b-9bf7-72eb98018ba4, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 5/5 exited 1, no more retries left.
Job chaiml-llama31-mer-v2-44570-v78-uploader completed after 41.12s with status: failed
Job failed chaiml-llama31-mer-v2-44570-v78-uploader:
Stopping job with name chaiml-llama31-mer-v2-44570-v78-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-llama31-mer-v2-44570-v78-uploader
Waiting for job on chaiml-llama31-mer-v2-44570-v78-uploader to finish
chaiml-llama31-mer-v2-44570-v78-uploader: Using quantization_mode: none
chaiml-llama31-mer-v2-44570-v78-uploader: Downloading snapshot of ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572...
chaiml-llama31-mer-v2-44570-v78-uploader: Downloaded in 5.769s
chaiml-llama31-mer-v2-44570-v78-uploader: Processed model ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572 in 11.447s
chaiml-llama31-mer-v2-44570-v78-uploader: creating bucket guanaco-vllm-models
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-llama31-mer-v2-44570-v78-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v78-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-llama31-mer-v2-44570-v78-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-llama31-mer-v2-44570-v78-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-llama31-mer-v2-44570-v78-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-llama31-mer-v2-44570-v78-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-llama31-mer-v2-44570-v78-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: a57eb2fc-b25a-9ee3-8046-de435dfefd54, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 38a71afe-292e-9075-aabe-92ffcf42e156, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: a7af05da-30d7-9518-96ce-85630611257f, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: ef2e4943-3a1b-974b-af16-528ce647ea0c, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 0cfee4e2-7193-9399-ac8f-701bf3c0d334, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 8794db5a-d4b2-9fe4-8099-ff9a96efacac, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 3b282697-1131-951f-b06a-9a1294b542da, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: ad97ed70-fd0b-9119-a2fc-c62806e0d614, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 2b1a9e94-4581-917b-ad34-0ba5b9525908, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: cdcc7506-1f72-9874-a2f1-46c89c09b717, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 3c6a4b28-9d9e-967d-912d-aaac3dd43f71, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 6e378d5d-22f4-92f2-9897-809d5113f36c, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: d128a956-fc39-9afb-8dbb-dcfac94644c0, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 4/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR session: fetching region failed: BadRequest: Bad Request
chaiml-llama31-mer-v2-44570-v78-uploader: status code: 400, request id: 603ba4bb-8b66-9987-a965-61efabc6b654, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/.gitattributes": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 54c67e3f-404f-990d-b52d-6661ff349a72, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: ada5b8cb-c50d-9614-a26e-ef9b21f79494, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/README.md": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: d3b56451-c90e-9c44-b8f7-84f246b8d43d, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00001-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 5cf45714-3bae-90dc-b4b5-fc56e585b7bf, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00003-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: f35cb771-8071-9140-9412-ccc114f19776, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00004-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: b7add6fa-6ec8-9d24-ba61-682053354c6d, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model-00002-of-00004.safetensors": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 932194e3-2629-9968-a8db-7204a5ef7022, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer_config.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 2afb32b9-95a1-91a8-9820-393f4293eaa5, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/model.safetensors.index.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: 4e014ed3-b5d8-952c-9f07-29353ffedea7, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/special_tokens_map.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: d7d56743-2551-9ac0-968e-5bd70145693b, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: ERROR "cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v78/default/tokenizer.json": PathStyleRequestNotAllowed: The path style requests are not allowed for this method, please switch to hostname-based requests. status code: 400, request id: c82d345e-378c-96c4-9a10-d0b7a87d9495, host id:
chaiml-llama31-mer-v2-44570-v78-uploader: Retry 5/5 exited 1, no more retries left.
Job chaiml-llama31-mer-v2-44570-v78-uploader completed after 42.59s with status: failed
Job failed chaiml-llama31-mer-v2-44570-v78-uploader:
Stopping job with name chaiml-llama31-mer-v2-44570-v78-uploader
clean up pipeline due to error=VLLMUploaderError('')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.54s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Pipeline stage VLLMModelDeleter completed in 0.53s
Shutdown handler de-registered
chaiml-llama31-mer-v2-_44570_v78 status is now failed due to DeploymentManager action
chaiml-llama31-mer-v2-_44570_v78 status is now torndown due to DeploymentManager action