chaiml-llama31-mer-v2-_44570

submission_id: chaiml-llama31-mer-v2-_44570_v77
developer_uid: chai_backend_admin
status: torndown
model_repo: ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': True}
timestamp: 2026-02-16T19:51:09+00:00
model_name: chaiml-llama31-mer-v2-_44570_v77
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-llama31-mer-v2-44570-v77-uploader
Waiting for job on chaiml-llama31-mer-v2-44570-v77-uploader to finish
chaiml-llama31-mer-v2-44570-v77-uploader: Using quantization_mode: none
chaiml-llama31-mer-v2-44570-v77-uploader: Downloading snapshot of ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572...
chaiml-llama31-mer-v2-44570-v77-uploader: Processed model ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572 in 13.105s
chaiml-llama31-mer-v2-44570-v77-uploader: creating bucket guanaco-vllm-models
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-llama31-mer-v2-44570-v77-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-llama31-mer-v2-44570-v77-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-llama31-mer-v2-44570-v77-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-llama31-mer-v2-44570-v77-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-llama31-mer-v2-44570-v77-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 67d5b621-569f-97bb-82d5-1547a5bf926d, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 83fc9f18-2d3d-9b54-b37b-8b33ed652dbd, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 2/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: ddb0ea4e-88d9-973e-8233-1db203c20dc8, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 3/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 5b4e23cc-3a9f-984e-ae7b-9e170fd4c7d6, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 5/5 exited 1, no more retries left.
Job chaiml-llama31-mer-v2-44570-v77-uploader completed after 46.04s with status: failed
Job failed chaiml-llama31-mer-v2-44570-v77-uploader: 
Stopping job with name chaiml-llama31-mer-v2-44570-v77-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-llama31-mer-v2-44570-v77-uploader
Waiting for job on chaiml-llama31-mer-v2-44570-v77-uploader to finish
chaiml-llama31-mer-v2-44570-v77-uploader: Using quantization_mode: none
chaiml-llama31-mer-v2-44570-v77-uploader: Downloading snapshot of ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572...
chaiml-llama31-mer-v2-44570-v77-uploader: Downloaded in 6.004s
chaiml-llama31-mer-v2-44570-v77-uploader: Processed model ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572 in 11.586s
chaiml-llama31-mer-v2-44570-v77-uploader: creating bucket guanaco-vllm-models
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-llama31-mer-v2-44570-v77-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-llama31-mer-v2-44570-v77-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-llama31-mer-v2-44570-v77-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-llama31-mer-v2-44570-v77-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-llama31-mer-v2-44570-v77-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 1ca547c0-dff8-97d7-a33b-b42a1a569c4a, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 176ba96f-9311-976d-bc5e-8d02cf89dde7, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 2/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 764ccf59-8586-99dd-b5fb-85d82322a7bd, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 3/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: afaa2409-b183-9e53-87bb-191408cd972d, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 4/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 37b37afd-ceeb-926a-a165-0a11c27439db, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 5/5 exited 1, no more retries left.
Job chaiml-llama31-mer-v2-44570-v77-uploader completed after 36.2s with status: failed
Job failed chaiml-llama31-mer-v2-44570-v77-uploader: 
Stopping job with name chaiml-llama31-mer-v2-44570-v77-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-llama31-mer-v2-44570-v77-uploader
Waiting for job on chaiml-llama31-mer-v2-44570-v77-uploader to finish
chaiml-llama31-mer-v2-44570-v77-uploader: Using quantization_mode: none
chaiml-llama31-mer-v2-44570-v77-uploader: Downloading snapshot of ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572...
chaiml-llama31-mer-v2-44570-v77-uploader: Downloaded in 5.962s
chaiml-llama31-mer-v2-44570-v77-uploader: Processed model ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572 in 11.496s
chaiml-llama31-mer-v2-44570-v77-uploader: creating bucket guanaco-vllm-models
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-llama31-mer-v2-44570-v77-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-llama31-mer-v2-44570-v77-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-llama31-mer-v2-44570-v77-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-llama31-mer-v2-44570-v77-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-llama31-mer-v2-44570-v77-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-llama31-mer-v2-44570-v77-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-llama31-mer-v2-44570-v77-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 9ed37d11-4969-968d-8635-78d967dff306, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 33087c21-f3d8-9ea4-aba7-8c714a1ab68e, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 4/5 exited 1, retrying in 2 seconds...
chaiml-llama31-mer-v2-44570-v77-uploader: ERROR "sync --size-only=true --concurrency=4 --exclude=*/.cache/* /dev/shm/model_output/* s3://guanaco-vllm-models/chaiml-llama31-mer-v2-44570-v77/default/": NotFound: Not Found status code: 404, request id: 15ca8e56-0a27-9da9-9f17-c9cb93271769, host id:
chaiml-llama31-mer-v2-44570-v77-uploader: Retry 5/5 exited 1, no more retries left.
Job chaiml-llama31-mer-v2-44570-v77-uploader completed after 36.33s with status: failed
Job failed chaiml-llama31-mer-v2-44570-v77-uploader: 
Stopping job with name chaiml-llama31-mer-v2-44570-v77-uploader
clean up pipeline due to error=VLLMUploaderError('')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.49s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Pipeline stage VLLMModelDeleter completed in 0.53s
Shutdown handler de-registered
chaiml-llama31-mer-v2-_44570_v77 status is now failed due to DeploymentManager action
chaiml-llama31-mer-v2-_44570_v77 status is now torndown due to DeploymentManager action