Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v25-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v25-uploader to finish
chaiml-kimid-v9-opusdv-23365-v25-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v25-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v9-opusdv-23365-v25-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v25-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
chaiml-kimid-v9-opusdv-23365-v25-uploader: Downloaded in 148.174s
chaiml-kimid-v9-opusdv-23365-v25-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 148.719s
chaiml-kimid-v9-opusdv-23365-v25-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-23365-v25-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v25-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v9-opusdv-23365-v25-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v9-opusdv-23365-v25-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v9-opusdv-23365-v25-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v25-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v25-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v25-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v25-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v25-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v25-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v25-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v25-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v9-opusdv-23365-v25-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v25-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v9-opusdv-23365-v25-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v9-opusdv-23365-v25-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-23365-v25-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/added_tokens.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/config.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/tokenizer_config.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/.gitattributes
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/special_tokens_map.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/quantization_config.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/merges.txt
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/chat_template.jinja
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/generation_config.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/vocab.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model.safetensors.index.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/tokenizer.json
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00027-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v25-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v25/default/model-00006-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-23365-v25-uploader completed after 236.05s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v25-uploader
Pipeline stage VLLMUploader completed in 236.66s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.73s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v25
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v25 to be ready
Inference service chaiml-kimid-v9-opusdv-23365-v25 ready after 341.37788558006287s
Pipeline stage VLLMDeployer completed in 341.78s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.365358829498291s
Received healthy response to inference request in 2.107930898666382s
Received healthy response to inference request in 2.285848617553711s
Received healthy response to inference request in 2.4791769981384277s
Received healthy response to inference request in 1.8934624195098877s
Received healthy response to inference request in 2.0381906032562256s
Received healthy response to inference request in 1.9648020267486572s
Received healthy response to inference request in 2.381765842437744s
Received healthy response to inference request in 2.232452869415283s
Received healthy response to inference request in 1.9151959419250488s
Received healthy response to inference request in 2.1753556728363037s
Received healthy response to inference request in 2.2618014812469482s
Received healthy response to inference request in 3.0810296535491943s
Received healthy response to inference request in 2.4384799003601074s
Received healthy response to inference request in 1.9398536682128906s
Received healthy response to inference request in 2.4051172733306885s
Received healthy response to inference request in 2.341000556945801s
Received healthy response to inference request in 2.175976514816284s
Received healthy response to inference request in 2.383962392807007s
Received healthy response to inference request in 2.0091373920440674s
Received healthy response to inference request in 2.675260305404663s
Received healthy response to inference request in 1.9409904479980469s
Received healthy response to inference request in 2.073350667953491s
Received healthy response to inference request in 2.316399574279785s
Received healthy response to inference request in 2.944772481918335s
Received healthy response to inference request in 1.9304900169372559s
Received healthy response to inference request in 2.2998673915863037s
Received healthy response to inference request in 2.8084568977355957s
Received healthy response to inference request in 2.0503273010253906s
Received healthy response to inference request in 2.1114964485168457s
30 requests
0 failed requests
5th percentile: 1.922078275680542
10th percentile: 1.938917303085327
20th percentile: 2.0002703189849855
30th percentile: 2.066443657875061
40th percentile: 2.1498119831085205
50th percentile: 2.2471271753311157
60th percentile: 2.3064802646636964
70th percentile: 2.3702809333801267
80th percentile: 2.411789798736572
90th percentile: 2.6885799646377566
95th percentile: 2.883430469036102
99th percentile: 3.041515073776245
mean time: 2.267577036221822
Pipeline stage StressChecker completed in 73.76s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.77s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_23365_v25 status is now deployed due to DeploymentManager action
chaiml-kimid-v9-opusdv_23365_v25 status is now inactive due to system request