Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-bol-opusd-v7-lr5-65112-v2-uploader
Waiting for job on chaiml-bol-opusd-v7-lr5-65112-v2-uploader to finish
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: Checking if ChaiML/bol-opusd-v7-lr5e6ep2r64g4b02-W4A16 already exists in ChaiML
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: Downloading snapshot of ChaiML/bol-opusd-v7-lr5e6ep2r64g4b02-W4A16...
chaiml-bol-opusd-v7-lr5-65112-v2-uploader:
Fetching 39 files: 0%| | 0/39 [00:00<?, ?it/s]
Fetching 39 files: 3%|▎ | 1/39 [00:00<00:09, 4.04it/s]
Fetching 39 files: 18%|█▊ | 7/39 [00:14<01:07, 2.12s/it]
Fetching 39 files: 21%|██ | 8/39 [00:17<01:10, 2.26s/it]
Fetching 39 files: 31%|███ | 12/39 [00:17<00:31, 1.16s/it]
Fetching 39 files: 33%|███▎ | 13/39 [00:17<00:26, 1.03s/it]
Fetching 39 files: 38%|███▊ | 15/39 [00:30<01:01, 2.58s/it]
Fetching 39 files: 46%|████▌ | 18/39 [00:31<00:36, 1.76s/it]
Fetching 39 files: 49%|████▊ | 19/39 [00:33<00:35, 1.75s/it]
Fetching 39 files: 51%|█████▏ | 20/39 [00:34<00:31, 1.64s/it]
Fetching 39 files: 54%|█████▍ | 21/39 [00:34<00:25, 1.41s/it]
Fetching 39 files: 59%|█████▉ | 23/39 [00:42<00:37, 2.33s/it]
Fetching 39 files: 62%|██████▏ | 24/39 [00:46<00:40, 2.69s/it]
Fetching 39 files: 67%|██████▋ | 26/39 [00:47<00:22, 1.75s/it]
Fetching 39 files: 69%|██████▉ | 27/39 [00:47<00:18, 1.56s/it]
Fetching 39 files: 72%|███████▏ | 28/39 [00:50<00:18, 1.72s/it]
Fetching 39 files: 79%|███████▉ | 31/39 [00:52<00:10, 1.27s/it]
Fetching 39 files: 82%|████████▏ | 32/39 [00:53<00:08, 1.19s/it]
Fetching 39 files: 100%|██████████| 39/39 [00:53<00:00, 1.37s/it]
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: Downloaded in 53.589s
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: Processed model ChaiML/bol-opusd-v7-lr5e6ep2r64g4b02 in 54.212s
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: creating bucket guanaco-vllm-models
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/chat_template.jinja
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/generation_config.json
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/added_tokens.json
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/.gitattributes
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/special_tokens_map.json
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/tokenizer_config.json
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/config.json
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/quantization_config.json
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/vocab.json
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/merges.txt
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/tokenizer.json
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model.safetensors.index.json
HTTP Request: %s %s "%s %d %s"
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00001-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00013-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00017-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00024-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00006-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00026-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00018-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00009-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00008-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00005-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00022-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00025-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00014-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00011-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00019-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00021-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00003-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00016-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00002-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00010-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00012-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00015-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00023-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00004-of-00027.safetensors
chaiml-bol-opusd-v7-lr5-65112-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-opusd-v7-lr5-65112-v2/model-00007-of-00027.safetensors
Job chaiml-bol-opusd-v7-lr5-65112-v2-uploader completed after 493.07s with status: succeeded
Stopping job with name chaiml-bol-opusd-v7-lr5-65112-v2-uploader
Pipeline stage VLLMUploader completed in 494.72s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.20s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-bol-opusd-v7-lr5-65112-v2
Waiting for inference service chaiml-bol-opusd-v7-lr5-65112-v2 to be ready
Inference service chaiml-bol-opusd-v7-lr5-65112-v2 ready after 540.5639200210571s
Pipeline stage VLLMDeployer completed in 540.93s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.042269468307495s
Received healthy response to inference request in 2.153515338897705s
Received healthy response to inference request in 2.029839515686035s
Received healthy response to inference request in 1.8520987033843994s
Received healthy response to inference request in 1.9120886325836182s
Received healthy response to inference request in 1.9367742538452148s
Received healthy response to inference request in 1.8575851917266846s
Received healthy response to inference request in 1.9488670825958252s
Received healthy response to inference request in 1.8590903282165527s
Received healthy response to inference request in 1.9733476638793945s
Received healthy response to inference request in 2.0163817405700684s
Received healthy response to inference request in 2.0938563346862793s
Received healthy response to inference request in 1.990962028503418s
Received healthy response to inference request in 1.850600242614746s
Received healthy response to inference request in 1.8908500671386719s
Received healthy response to inference request in 2.24568510055542s
Received healthy response to inference request in 1.901644229888916s
Received healthy response to inference request in 2.0852880477905273s
Received healthy response to inference request in 1.9471135139465332s
Received healthy response to inference request in 2.20491361618042s
Received healthy response to inference request in 2.1379811763763428s
Received healthy response to inference request in 1.8520574569702148s
Received healthy response to inference request in 1.8931000232696533s
Received healthy response to inference request in 2.1122725009918213s
Received healthy response to inference request in 2.0890536308288574s
Received healthy response to inference request in 2.157339572906494s
Received healthy response to inference request in 1.8677234649658203s
Received healthy response to inference request in 2.046872854232788s
Received healthy response to inference request in 1.9702115058898926s
Received healthy response to inference request in 2.0039279460906982s
30 requests
0 failed requests
5th percentile: 1.8520760178565978
10th percentile: 1.857036542892456
20th percentile: 1.8862247467041016
30th percentile: 1.9089553117752076
40th percentile: 1.9481656551361084
50th percentile: 1.9821548461914062
60th percentile: 2.021764850616455
70th percentile: 2.05839741230011
80th percentile: 2.0975395679473876
90th percentile: 2.153897762298584
95th percentile: 2.1835052967071533
99th percentile: 2.23386137008667
mean time: 1.997443707784017
Pipeline stage StressChecker completed in 64.61s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.67s
Shutdown handler de-registered
chaiml-bol-opusd-v7-lr5_65112_v2 status is now deployed due to DeploymentManager action
chaiml-bol-opusd-v7-lr5_65112_v2 status is now inactive due to auto deactivation removed underperforming models