Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-bol-opusd-v7-lr5-65112-v5-uploader
Waiting for job on chaiml-bol-opusd-v7-lr5-65112-v5-uploader to finish
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: Using quantization_mode: w4a16
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: Checking if ChaiML/bol-opusd-v7-lr5e6ep2r64g4b02-W4A16 already exists in ChaiML
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: Downloading snapshot of ChaiML/bol-opusd-v7-lr5e6ep2r64g4b02-W4A16...
chaiml-bol-opusd-v7-lr5-65112-v5-uploader:
Fetching 39 files: 0%| | 0/39 [00:00<?, ?it/s]
Fetching 39 files: 3%|▎ | 1/39 [00:00<00:08, 4.29it/s]
Fetching 39 files: 18%|█▊ | 7/39 [00:13<01:03, 1.98s/it]
Fetching 39 files: 23%|██▎ | 9/39 [00:14<00:46, 1.54s/it]
Fetching 39 files: 31%|███ | 12/39 [00:14<00:26, 1.03it/s]
Fetching 39 files: 33%|███▎ | 13/39 [00:14<00:22, 1.17it/s]
Fetching 39 files: 38%|███▊ | 15/39 [00:26<00:56, 2.36s/it]
Fetching 39 files: 46%|████▌ | 18/39 [00:26<00:30, 1.44s/it]
Fetching 39 files: 51%|█████▏ | 20/39 [00:26<00:20, 1.06s/it]
Fetching 39 files: 56%|█████▋ | 22/39 [00:27<00:15, 1.13it/s]
Fetching 39 files: 59%|█████▉ | 23/39 [00:38<00:40, 2.53s/it]
Fetching 39 files: 64%|██████▍ | 25/39 [00:39<00:27, 1.97s/it]
Fetching 39 files: 69%|██████▉ | 27/39 [00:40<00:16, 1.41s/it]
Fetching 39 files: 74%|███████▍ | 29/39 [00:41<00:11, 1.11s/it]
Fetching 39 files: 79%|███████▉ | 31/39 [00:44<00:10, 1.28s/it]
Fetching 39 files: 82%|████████▏ | 32/39 [00:45<00:08, 1.17s/it]
Fetching 39 files: 100%|██████████| 39/39 [00:45<00:00, 1.15s/it]
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: Downloaded in 45.098s
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: Processed model ChaiML/bol-opusd-v7-lr5e6ep2r64g4b02 in 45.769s
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: creating bucket guanaco-vllm-models
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/added_tokens.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/chat_template.jinja
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/generation_config.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/.gitattributes
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/quantization_config.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/merges.txt
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/tokenizer_config.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/special_tokens_map.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/vocab.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/config.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model.safetensors.index.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/tokenizer.json
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00027-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00003-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00022-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00005-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00009-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00020-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00016-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00008-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00013-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00011-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00021-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00002-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00026-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00007-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00017-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00025-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00014-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00024-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00004-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00006-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00018-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00019-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00015-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00012-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00023-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v5-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v5/model-00010-of-00027.safetensors
Job chaiml-bol-opusd-v7-lr5-65112-v5-uploader completed after 800.39s with status: succeeded
Stopping job with name chaiml-bol-opusd-v7-lr5-65112-v5-uploader
Pipeline stage VLLMUploader completed in 800.77s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-bol-opusd-v7-lr5-65112-v5
Waiting for inference service chaiml-bol-opusd-v7-lr5-65112-v5 to be ready
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-bol-opusd-v7-lr5-65112-v5 ready after 671.9719748497009s
Pipeline stage VLLMDeployer completed in 672.43s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1513404846191406s
Received healthy response to inference request in 1.673774003982544s
Received healthy response to inference request in 2.0092217922210693s
Received healthy response to inference request in 1.687565803527832s
Received healthy response to inference request in 1.694667100906372s
Received healthy response to inference request in 1.7006640434265137s
Received healthy response to inference request in 1.770334243774414s
Received healthy response to inference request in 2.0510926246643066s
Received healthy response to inference request in 1.6938011646270752s
Received healthy response to inference request in 1.8220844268798828s
Received healthy response to inference request in 1.793421983718872s
Received healthy response to inference request in 2.0640764236450195s
Received healthy response to inference request in 1.828934669494629s
Received healthy response to inference request in 2.14237117767334s
Received healthy response to inference request in 2.044994592666626s
Received healthy response to inference request in 1.7319388389587402s
Received healthy response to inference request in 2.0857200622558594s
Received healthy response to inference request in 1.9545972347259521s
Received healthy response to inference request in 1.7220265865325928s
Received healthy response to inference request in 1.6940534114837646s
Received healthy response to inference request in 1.854738473892212s
Received healthy response to inference request in 1.6829121112823486s
Received healthy response to inference request in 1.7644846439361572s
Received healthy response to inference request in 2.088479995727539s
Received healthy response to inference request in 1.959496259689331s
Received healthy response to inference request in 1.846792459487915s
Received healthy response to inference request in 1.7949707508087158s
Received healthy response to inference request in 1.7933762073516846s
Received healthy response to inference request in 2.042239189147949s
Received healthy response to inference request in 1.959820032119751s
30 requests
0 failed requests
5th percentile: 1.6850062727928161
10th percentile: 1.693177628517151
20th percentile: 1.6994646549224854
30th percentile: 1.754720902442932
40th percentile: 1.7934036731719971
50th percentile: 1.8255095481872559
60th percentile: 1.8946819782257078
70th percentile: 1.9746405601501464
80th percentile: 2.046214199066162
90th percentile: 2.0859960556030273
95th percentile: 2.1181201457977292
99th percentile: 2.148739385604858
mean time: 1.8701330264409384
Pipeline stage StressChecker completed in 59.06s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.87s
Shutdown handler de-registered
chaiml-bol-opusd-v7-lr5_65112_v5 status is now deployed due to DeploymentManager action
chaiml-bol-opusd-v7-lr5_65112_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-bol-opusd-v7-lr5_65112_v5 status is now torndown due to DeploymentManager action