Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-opusd-v4-g46-lr1-17132-v1-uploader
Waiting for job on chaiml-opusd-v4-g46-lr1-17132-v1-uploader to finish
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: Using quantization_mode: none
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: Downloading snapshot of ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8...
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: Downloaded in 159.620s
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: Processed model ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8 in 291.288s
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: creating bucket guanaco-vllm-models
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/.gitattributes
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/recipe.yaml
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/generation_config.json
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/tokenizer_config.json
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/special_tokens_map.json
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/tokenizer.json
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/config.json
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/chat_template.jinja
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model.safetensors.index.json
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00072-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00071-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00013-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00058-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00005-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00065-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00038-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00011-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00017-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00067-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00015-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00050-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00046-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00021-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00029-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00064-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00041-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00035-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00054-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00049-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00008-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00044-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00053-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00060-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00022-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00036-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00070-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00039-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00063-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00037-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00048-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00028-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00066-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00018-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00061-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00052-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00057-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00002-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00027-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00047-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00042-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00059-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00032-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00062-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00014-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00019-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00030-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00033-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00012-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00034-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00004-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00045-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00024-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00006-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00040-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00031-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00068-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00003-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00001-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00051-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00069-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00055-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00007-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00026-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00043-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00020-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00023-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00010-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00056-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00025-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00009-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v1/default/model-00016-of-00072.safetensors
Job chaiml-opusd-v4-g46-lr1-17132-v1-uploader completed after 403.85s with status: succeeded
Stopping job with name chaiml-opusd-v4-g46-lr1-17132-v1-uploader
Pipeline stage VLLMUploader completed in 404.69s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.24s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-opusd-v4-g46-lr1-17132-v1
Waiting for inference service chaiml-opusd-v4-g46-lr1-17132-v1 to be ready
Inference service chaiml-opusd-v4-g46-lr1-17132-v1 ready after 1631.6512835025787s
Pipeline stage VLLMDeployer completed in 1632.04s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9708189964294434s
Received healthy response to inference request in 1.965043067932129s
Received healthy response to inference request in 4.228227615356445s
Received healthy response to inference request in 1.8523283004760742s
Received healthy response to inference request in 1.9458098411560059s
Received healthy response to inference request in 1.8544034957885742s
Received healthy response to inference request in 2.0199546813964844s
Received healthy response to inference request in 1.8660995960235596s
Received healthy response to inference request in 1.9160106182098389s
Received healthy response to inference request in 1.9220125675201416s
Received healthy response to inference request in 1.9807524681091309s
Received healthy response to inference request in 2.0842645168304443s
Received healthy response to inference request in 1.89951753616333s
Received healthy response to inference request in 1.939471960067749s
Received healthy response to inference request in 2.0907583236694336s
Received healthy response to inference request in 2.066298007965088s
Received healthy response to inference request in 2.06022047996521s
Received healthy response to inference request in 2.0716474056243896s
Received healthy response to inference request in 1.9257171154022217s
Received healthy response to inference request in 1.9538068771362305s
Received healthy response to inference request in 1.9914557933807373s
Received healthy response to inference request in 1.9310846328735352s
Received healthy response to inference request in 2.244356632232666s
Received healthy response to inference request in 1.9644381999969482s
Received healthy response to inference request in 1.995093822479248s
Received healthy response to inference request in 2.1035165786743164s
Received healthy response to inference request in 1.969709873199463s
Received healthy response to inference request in 2.0648715496063232s
Received healthy response to inference request in 1.9437005519866943s
Received healthy response to inference request in 2.054232120513916s
30 requests
0 failed requests
5th percentile: 1.8596667408943177
10th percentile: 1.896175742149353
20th percentile: 1.9249762058258058
30th percentile: 1.9424319744110108
40th percentile: 1.960185670852661
50th percentile: 1.9702644348144531
60th percentile: 1.9929110050201415
70th percentile: 2.056028628349304
80th percentile: 2.0673678874969483
90th percentile: 2.0920341491699217
95th percentile: 2.1809786081314084
99th percentile: 3.652905030250551
mean time: 2.0625207742055256
Pipeline stage StressChecker completed in 65.84s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.07s
Shutdown handler de-registered
chaiml-opusd-v4-g46-lr1_17132_v1 status is now deployed due to DeploymentManager action
chaiml-opusd-v4-g46-lr1_17132_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-opusd-v4-g46-lr1_17132_v1 status is now torndown due to DeploymentManager action