Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-bol-v6-opusdv1b-lr-759-v3-uploader
Waiting for job on chaiml-bol-v6-opusdv1b-lr-759-v3-uploader to finish
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: Using quantization_mode: w4a16
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: Checking if ChaiML/bol-v6-opusdv1b-lr1e5ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: Downloading snapshot of ChaiML/bol-v6-opusdv1b-lr1e5ep2r64g4b01-W4A16...
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader:
Fetching 39 files: 0%| | 0/39 [00:00<?, ?it/s]
Fetching 39 files: 3%|▎ | 1/39 [00:00<00:12, 2.93it/s]
Fetching 39 files: 18%|█▊ | 7/39 [00:15<01:11, 2.23s/it]
Fetching 39 files: 21%|██ | 8/39 [00:18<01:17, 2.51s/it]
Fetching 39 files: 26%|██▌ | 10/39 [00:19<00:49, 1.70s/it]
Fetching 39 files: 38%|███▊ | 15/39 [00:28<00:43, 1.83s/it]
Fetching 39 files: 41%|████ | 16/39 [00:30<00:41, 1.79s/it]
Fetching 39 files: 44%|████▎ | 17/39 [00:32<00:42, 1.92s/it]
Fetching 39 files: 46%|████▌ | 18/39 [00:35<00:41, 1.97s/it]
Fetching 39 files: 51%|█████▏ | 20/39 [00:36<00:28, 1.48s/it]
Fetching 39 files: 59%|█████▉ | 23/39 [00:44<00:31, 1.97s/it]
Fetching 39 files: 62%|██████▏ | 24/39 [00:45<00:27, 1.85s/it]
Fetching 39 files: 64%|██████▍ | 25/39 [00:48<00:29, 2.08s/it]
Fetching 39 files: 67%|██████▋ | 26/39 [00:51<00:28, 2.23s/it]
Fetching 39 files: 69%|██████▉ | 27/39 [00:52<00:23, 1.93s/it]
Fetching 39 files: 74%|███████▍ | 29/39 [00:53<00:13, 1.37s/it]
Fetching 39 files: 79%|███████▉ | 31/39 [00:55<00:09, 1.24s/it]
Fetching 39 files: 82%|████████▏ | 32/39 [00:56<00:08, 1.15s/it]
Fetching 39 files: 100%|██████████| 39/39 [00:56<00:00, 1.44s/it]
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: Downloaded in 56.099s
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: Processed model ChaiML/bol-v6-opusdv1b-lr1e5ep2r64g4b01 in 56.763s
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: creating bucket guanaco-vllm-models
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/.gitattributes
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/generation_config.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/added_tokens.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/special_tokens_map.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/chat_template.jinja
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/quantization_config.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/tokenizer_config.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/merges.txt
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/vocab.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/config.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model.safetensors.index.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/tokenizer.json
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00027-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00002-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00011-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00004-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00025-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00015-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00012-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00006-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00023-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00008-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00009-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00014-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00018-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00026-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00001-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00024-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00019-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00020-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00013-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00007-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00022-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00017-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00003-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00010-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00005-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00021-of-00027.safetensors
chaiml-bol-v6-opusdv1b-lr-759-v3-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v6-opusdv1b-lr-759-v3/model-00016-of-00027.safetensors
Job chaiml-bol-v6-opusdv1b-lr-759-v3-uploader completed after 452.65s with status: succeeded
Stopping job with name chaiml-bol-v6-opusdv1b-lr-759-v3-uploader
Pipeline stage VLLMUploader completed in 453.02s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-bol-v6-opusdv1b-lr-759-v3
Waiting for inference service chaiml-bol-v6-opusdv1b-lr-759-v3 to be ready
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-bol-v6-opusdv1b-lr-759-v3 ready after 590.3694376945496s
Pipeline stage VLLMDeployer completed in 590.72s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.8771896362304688s
Received healthy response to inference request in 1.763911485671997s
Received healthy response to inference request in 1.9911606311798096s
Received healthy response to inference request in 1.6769802570343018s
Received healthy response to inference request in 1.7283101081848145s
Received healthy response to inference request in 2.142780065536499s
Received healthy response to inference request in 1.739222764968872s
Received healthy response to inference request in 1.8181638717651367s
Received healthy response to inference request in 1.8389592170715332s
Received healthy response to inference request in 1.839644432067871s
Received healthy response to inference request in 1.7190005779266357s
Received healthy response to inference request in 1.856398582458496s
Received healthy response to inference request in 1.759742021560669s
Received healthy response to inference request in 1.8275420665740967s
Received healthy response to inference request in 1.9349489212036133s
Received healthy response to inference request in 1.806016206741333s
Received healthy response to inference request in 2.1888890266418457s
Received healthy response to inference request in 1.9311785697937012s
Received healthy response to inference request in 1.8548483848571777s
Received healthy response to inference request in 1.9153671264648438s
Received healthy response to inference request in 1.7610552310943604s
Received healthy response to inference request in 1.7515459060668945s
Received healthy response to inference request in 1.8345146179199219s
Received healthy response to inference request in 1.7902672290802002s
Received healthy response to inference request in 1.9934864044189453s
Received healthy response to inference request in 1.7352778911590576s
Received healthy response to inference request in 1.8481802940368652s
Received healthy response to inference request in 2.0672860145568848s
Received healthy response to inference request in 1.738309383392334s
Received healthy response to inference request in 1.8140451908111572s
30 requests
0 failed requests
5th percentile: 1.7231898665428163
10th percentile: 1.7345811128616333
20th percentile: 1.74908127784729
30th percentile: 1.763054609298706
40th percentile: 1.8108335971832275
50th percentile: 1.8310283422470093
60th percentile: 1.8430587768554687
70th percentile: 1.862635898590088
80th percentile: 1.9319326400756835
90th percentile: 2.0008663654327394
95th percentile: 2.1088077425956726
99th percentile: 2.175517427921295
mean time: 1.8514740705490111
Pipeline stage StressChecker completed in 71.89s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.90s
Shutdown handler de-registered
chaiml-bol-v6-opusdv1b-lr_759_v3 status is now deployed due to DeploymentManager action
chaiml-bol-v6-opusdv1b-lr_759_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-bol-v6-opusdv1b-lr_759_v3 status is now torndown due to DeploymentManager action