Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-opusd-v4-g46-mas-11495-v5-uploader
Waiting for job on chaiml-opusd-v4-g46-mas-11495-v5-uploader to finish
chaiml-opusd-v4-g46-mas-11495-v5-uploader: Using quantization_mode: fp8
chaiml-opusd-v4-g46-mas-11495-v5-uploader: Checking if ChaiML/opusd-v4-g46-masked-think-lr1e4ep1r64b16-pack-FP8 already exists in ChaiML
chaiml-opusd-v4-g46-mas-11495-v5-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-opusd-v4-g46-mas-11495-v5-uploader: Downloading snapshot of ChaiML/opusd-v4-g46-masked-think-lr1e4ep1r64b16-pack-FP8...
chaiml-opusd-v4-g46-mas-11495-v5-uploader: Downloaded in 129.734s
chaiml-opusd-v4-g46-mas-11495-v5-uploader: Processed model ChaiML/opusd-v4-g46-masked-think-lr1e4ep1r64b16-pack in 133.152s
chaiml-opusd-v4-g46-mas-11495-v5-uploader: creating bucket guanaco-vllm-models
chaiml-opusd-v4-g46-mas-11495-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-mas-11495-v5-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-opusd-v4-g46-mas-11495-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-opusd-v4-g46-mas-11495-v5-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-opusd-v4-g46-mas-11495-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-mas-11495-v5-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-mas-11495-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-mas-11495-v5-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-mas-11495-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-mas-11495-v5-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-mas-11495-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-mas-11495-v5-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-mas-11495-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-opusd-v4-g46-mas-11495-v5-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-opusd-v4-g46-mas-11495-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-opusd-v4-g46-mas-11495-v5-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-opusd-v4-g46-mas-11495-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-opusd-v4-g46-mas-11495-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/.gitattributes
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/generation_config.json
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/config.json
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/chat_template.jinja
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/recipe.yaml
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/special_tokens_map.json
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/tokenizer_config.json
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model.safetensors.index.json
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/tokenizer.json
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00072-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00071-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00049-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00029-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00010-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00065-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00062-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00060-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00006-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00048-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00007-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00047-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00067-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00063-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00050-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00013-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00030-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00044-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00058-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00056-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00057-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00061-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00017-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00040-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00014-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00020-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00042-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00055-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00027-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00003-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00034-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00025-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00026-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00041-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00002-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00068-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00066-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00032-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00009-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00022-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00070-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00028-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00023-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00059-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00051-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00015-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00001-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00039-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00024-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00035-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00037-of-00072.safetensors
chaiml-opusd-v4-g46-mas-11495-v5-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-mas-11495-v5/default/model-00069-of-00072.safetensors
Job chaiml-opusd-v4-g46-mas-11495-v5-uploader completed after 268.75s with status: succeeded
Stopping job with name chaiml-opusd-v4-g46-mas-11495-v5-uploader
Pipeline stage VLLMUploader completed in 269.22s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.63s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-opusd-v4-g46-mas-11495-v5
Waiting for inference service chaiml-opusd-v4-g46-mas-11495-v5 to be ready
Inference service chaiml-opusd-v4-g46-mas-11495-v5 ready after 591.0726141929626s
Pipeline stage VLLMDeployer completed in 591.45s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.6560587882995605s
Received healthy response to inference request in 1.8313367366790771s
Received healthy response to inference request in 1.8407773971557617s
Received healthy response to inference request in 2.0232632160186768s
Received healthy response to inference request in 2.029390811920166s
Received healthy response to inference request in 1.9761593341827393s
Received healthy response to inference request in 1.8487129211425781s
Received healthy response to inference request in 1.979297161102295s
Received healthy response to inference request in 2.6101410388946533s
Received healthy response to inference request in 1.9937798976898193s
Received healthy response to inference request in 2.1486315727233887s
Received healthy response to inference request in 2.068648099899292s
Received healthy response to inference request in 2.086658000946045s
Received healthy response to inference request in 2.107235908508301s
Received healthy response to inference request in 2.2878305912017822s
Received healthy response to inference request in 2.114213705062866s
Received healthy response to inference request in 1.92153000831604s
Received healthy response to inference request in 1.8799326419830322s
Received healthy response to inference request in 1.8956832885742188s
Received healthy response to inference request in 2.786504030227661s
Received healthy response to inference request in 2.447662353515625s
Received healthy response to inference request in 2.402160882949829s
Received healthy response to inference request in 2.1129467487335205s
Received healthy response to inference request in 2.410731077194214s
Received healthy response to inference request in 1.9770190715789795s
Received healthy response to inference request in 2.3882174491882324s
Received healthy response to inference request in 2.0162689685821533s
Received healthy response to inference request in 2.770972967147827s
Received healthy response to inference request in 2.0285873413085938s
Received healthy response to inference request in 1.9588909149169922s
30 requests
0 failed requests
5th percentile: 1.844348382949829
10th percentile: 1.8768106698989868
20th percentile: 1.9514187335968018
30th percentile: 1.9786137342453003
40th percentile: 2.0204655170440673
50th percentile: 2.049019455909729
60th percentile: 2.1095202445983885
70th percentile: 2.1903912782669064
80th percentile: 2.403874921798706
90th percentile: 2.614732813835144
95th percentile: 2.7192615866661067
99th percentile: 2.7820000219345093
mean time: 2.1533080975214642
Pipeline stage StressChecker completed in 81.13s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.74s
Shutdown handler de-registered
chaiml-opusd-v4-g46-mas_11495_v5 status is now deployed due to DeploymentManager action
chaiml-opusd-v4-g46-mas_11495_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-opusd-v4-g46-mas_11495_v5 status is now torndown due to DeploymentManager action