Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-muster-v0d-lr1e5-93728-v5-uploader
Waiting for job on chaiml-muster-v0d-lr1e5-93728-v5-uploader to finish
chaiml-muster-v0d-lr1e5-93728-v5-uploader: Using quantization_mode: w4a16
chaiml-muster-v0d-lr1e5-93728-v5-uploader: Checking if ChaiML/muster-v0d-lr1e5ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-muster-v0d-lr1e5-93728-v5-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-muster-v0d-lr1e5-93728-v5-uploader: Downloading snapshot of ChaiML/muster-v0d-lr1e5ep2r64g4b01-W4A16...
chaiml-muster-v0d-lr1e5-93728-v5-uploader:
Fetching 39 files: 0%| | 0/39 [00:00<?, ?it/s]
Fetching 39 files: 3%|▎ | 1/39 [00:00<00:12, 3.16it/s]
Fetching 39 files: 18%|█▊ | 7/39 [00:15<01:15, 2.36s/it]
Fetching 39 files: 21%|██ | 8/39 [00:16<01:01, 1.99s/it]
Fetching 39 files: 26%|██▌ | 10/39 [00:18<00:49, 1.69s/it]
Fetching 39 files: 38%|███▊ | 15/39 [00:29<00:46, 1.95s/it]
Fetching 39 files: 41%|████ | 16/39 [00:29<00:40, 1.76s/it]
Fetching 39 files: 44%|████▎ | 17/39 [00:31<00:37, 1.69s/it]
Fetching 39 files: 49%|████▊ | 19/39 [00:31<00:25, 1.28s/it]
Fetching 39 files: 51%|█████▏ | 20/39 [00:32<00:21, 1.13s/it]
Fetching 39 files: 54%|█████▍ | 21/39 [00:33<00:20, 1.14s/it]
Fetching 39 files: 56%|█████▋ | 22/39 [00:33<00:16, 1.04it/s]
Fetching 39 files: 59%|█████▉ | 23/39 [00:44<00:53, 3.36s/it]
Fetching 39 files: 62%|██████▏ | 24/39 [00:44<00:38, 2.56s/it]
Fetching 39 files: 64%|██████▍ | 25/39 [00:46<00:31, 2.28s/it]
Fetching 39 files: 67%|██████▋ | 26/39 [00:46<00:22, 1.73s/it]
Fetching 39 files: 69%|██████▉ | 27/39 [00:47<00:16, 1.38s/it]
Fetching 39 files: 72%|███████▏ | 28/39 [00:47<00:12, 1.13s/it]
Fetching 39 files: 74%|███████▍ | 29/39 [00:48<00:11, 1.16s/it]
Fetching 39 files: 79%|███████▉ | 31/39 [00:51<00:10, 1.28s/it]
Fetching 39 files: 82%|████████▏ | 32/39 [00:52<00:07, 1.08s/it]
Fetching 39 files: 100%|██████████| 39/39 [00:52<00:00, 1.34s/it]
chaiml-muster-v0d-lr1e5-93728-v5-uploader: Downloaded in 52.250s
chaiml-muster-v0d-lr1e5-93728-v5-uploader: Processed model ChaiML/muster-v0d-lr1e5ep2r64g4b01 in 52.797s
chaiml-muster-v0d-lr1e5-93728-v5-uploader: creating bucket guanaco-vllm-models
chaiml-muster-v0d-lr1e5-93728-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0d-lr1e5-93728-v5-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-muster-v0d-lr1e5-93728-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-muster-v0d-lr1e5-93728-v5-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-muster-v0d-lr1e5-93728-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0d-lr1e5-93728-v5-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-muster-v0d-lr1e5-93728-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0d-lr1e5-93728-v5-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-muster-v0d-lr1e5-93728-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0d-lr1e5-93728-v5-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-muster-v0d-lr1e5-93728-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0d-lr1e5-93728-v5-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-muster-v0d-lr1e5-93728-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-muster-v0d-lr1e5-93728-v5-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-muster-v0d-lr1e5-93728-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-muster-v0d-lr1e5-93728-v5-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-muster-v0d-lr1e5-93728-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-muster-v0d-lr1e5-93728-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/.gitattributes
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/added_tokens.json
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/chat_template.jinja
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/generation_config.json
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/tokenizer_config.json
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/config.json
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/special_tokens_map.json
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/quantization_config.json
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/merges.txt
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/vocab.json
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model.safetensors.index.json
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/tokenizer.json
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00027-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00014-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00017-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00026-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00006-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00022-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00012-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00005-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00009-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00003-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00007-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00025-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00024-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00002-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00001-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00010-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00004-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00008-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00021-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00015-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00013-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00016-of-00027.safetensors
chaiml-muster-v0d-lr1e5-93728-v5-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0d-lr1e5-93728-v5/model-00019-of-00027.safetensors
Job chaiml-muster-v0d-lr1e5-93728-v5-uploader completed after 690.08s with status: succeeded
Stopping job with name chaiml-muster-v0d-lr1e5-93728-v5-uploader
Pipeline stage VLLMUploader completed in 690.44s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-muster-v0d-lr1e5-93728-v5
Waiting for inference service chaiml-muster-v0d-lr1e5-93728-v5 to be ready
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-muster-v0d-lr1e5-93728-v5 ready after 700.4095151424408s
Pipeline stage VLLMDeployer completed in 700.87s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2733731269836426s
Received healthy response to inference request in 1.9468393325805664s
Received healthy response to inference request in 1.9808754920959473s
Received healthy response to inference request in 2.2910940647125244s
Received healthy response to inference request in 1.935530185699463s
Received healthy response to inference request in 2.359884023666382s
Received healthy response to inference request in 1.9090118408203125s
Received healthy response to inference request in 2.0958993434906006s
Received healthy response to inference request in 2.941696882247925s
Received healthy response to inference request in 1.9663093090057373s
Received healthy response to inference request in 2.1570258140563965s
Received healthy response to inference request in 1.904677152633667s
Received healthy response to inference request in 2.0988779067993164s
Received healthy response to inference request in 2.1561288833618164s
Received healthy response to inference request in 2.0798213481903076s
Received healthy response to inference request in 1.9785363674163818s
Received healthy response to inference request in 2.29646635055542s
Received healthy response to inference request in 2.065181016921997s
Received healthy response to inference request in 2.044067144393921s
Received healthy response to inference request in 1.9154210090637207s
Received healthy response to inference request in 1.9383039474487305s
Received healthy response to inference request in 1.8962900638580322s
Received healthy response to inference request in 2.0120012760162354s
Received healthy response to inference request in 2.0744857788085938s
Received healthy response to inference request in 1.9581844806671143s
Received healthy response to inference request in 2.145293951034546s
Received healthy response to inference request in 1.961754560470581s
Received healthy response to inference request in 2.195868968963623s
Received healthy response to inference request in 2.437380790710449s
Received healthy response to inference request in 1.9541852474212646s
30 requests
0 failed requests
5th percentile: 1.9066277623176575
10th percentile: 1.9147800922393798
20th percentile: 1.9451322555541992
30th percentile: 1.960683536529541
40th percentile: 1.979939842224121
50th percentile: 2.054624080657959
60th percentile: 2.086252546310425
70th percentile: 2.148544430732727
80th percentile: 2.211369800567627
90th percentile: 2.3028081178665163
95th percentile: 2.402507245540619
99th percentile: 2.7954452157020575
mean time: 2.0990155220031737
Pipeline stage StressChecker completed in 68.10s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.64s
Shutdown handler de-registered
chaiml-muster-v0d-lr1e5_93728_v5 status is now deployed due to DeploymentManager action
chaiml-muster-v0d-lr1e5_93728_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-muster-v0d-lr1e5_93728_v5 status is now torndown due to DeploymentManager action