Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-1007-tl-ads-aggr-36615-v6-uploader
Waiting for job on chaiml-1007-tl-ads-aggr-36615-v6-uploader to finish
chaiml-1007-tl-ads-aggr-36615-v6-uploader: Using quantization_mode: fp8
chaiml-1007-tl-ads-aggr-36615-v6-uploader: Checking if ChaiML/1007-tl-ads-aggressive-gac1-FP8 already exists in ChaiML
chaiml-1007-tl-ads-aggr-36615-v6-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-1007-tl-ads-aggr-36615-v6-uploader: Downloading snapshot of ChaiML/1007-tl-ads-aggressive-gac1-FP8...
chaiml-1007-tl-ads-aggr-36615-v6-uploader: Downloaded in 8.000s
chaiml-1007-tl-ads-aggr-36615-v6-uploader: Processed model ChaiML/1007-tl-ads-aggressive-gac1 in 11.455s
chaiml-1007-tl-ads-aggr-36615-v6-uploader: creating bucket guanaco-vllm-models
chaiml-1007-tl-ads-aggr-36615-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v6-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-1007-tl-ads-aggr-36615-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-1007-tl-ads-aggr-36615-v6-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-1007-tl-ads-aggr-36615-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v6-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-aggr-36615-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v6-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-aggr-36615-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v6-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-aggr-36615-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v6-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-aggr-36615-v6-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-1007-tl-ads-aggr-36615-v6-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-1007-tl-ads-aggr-36615-v6-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-1007-tl-ads-aggr-36615-v6-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-1007-tl-ads-aggr-36615-v6-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-1007-tl-ads-aggr-36615-v6-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/special_tokens_map.json
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/generation_config.json
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/tokenizer_config.json
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/chat_template.jinja
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/.gitattributes
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/recipe.yaml
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/config.json
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/model.safetensors.index.json
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/tokenizer.json
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/model-00003-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/model-00003-of-00003.safetensors
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/model-00002-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/model-00002-of-00003.safetensors
chaiml-1007-tl-ads-aggr-36615-v6-uploader: cp /dev/shm/model_output/model-00001-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v6/default/model-00001-of-00003.safetensors
Job chaiml-1007-tl-ads-aggr-36615-v6-uploader completed after 52.2s with status: succeeded
Stopping job with name chaiml-1007-tl-ads-aggr-36615-v6-uploader
Pipeline stage VLLMUploader completed in 52.64s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.30s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-1007-tl-ads-aggr-36615-v6
Waiting for inference service chaiml-1007-tl-ads-aggr-36615-v6 to be ready
2026-03-20T23:04:12.959474+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v6
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-1007-tl-ads-run-2-gac-v7-uploader
Waiting for job on chaiml-1007-tl-ads-run-2-gac-v7-uploader to finish
chaiml-1007-tl-ads-run-2-gac-v7-uploader: Using quantization_mode: fp8
chaiml-1007-tl-ads-run-2-gac-v7-uploader: Checking if ChaiML/1007-tl-ads-run-2-gac-FP8 already exists in ChaiML
chaiml-1007-tl-ads-run-2-gac-v7-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-1007-tl-ads-run-2-gac-v7-uploader: Downloading snapshot of ChaiML/1007-tl-ads-run-2-gac-FP8...
2026-03-20T23:05:13.051022+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v6
2026-03-20T23:05:15.212389+00:00 monitor updated for chaiml-1007-tl-ads-run-2-gac_v7
chaiml-1007-tl-ads-run-2-gac-v7-uploader: creating bucket guanaco-vllm-models
chaiml-1007-tl-ads-run-2-gac-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-run-2-gac-v7-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-1007-tl-ads-run-2-gac-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-1007-tl-ads-run-2-gac-v7-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-1007-tl-ads-run-2-gac-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-run-2-gac-v7-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-run-2-gac-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-run-2-gac-v7-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-run-2-gac-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-run-2-gac-v7-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-run-2-gac-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-run-2-gac-v7-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-run-2-gac-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-1007-tl-ads-run-2-gac-v7-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-1007-tl-ads-run-2-gac-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-1007-tl-ads-run-2-gac-v7-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-1007-tl-ads-run-2-gac-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-1007-tl-ads-run-2-gac-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/recipe.yaml
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/config.json
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/chat_template.jinja
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/.gitattributes
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/special_tokens_map.json
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/model.safetensors.index.json
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/generation_config.json
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/tokenizer_config.json
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/tokenizer.json
chaiml-1007-tl-ads-run-2-gac-v7-uploader: cp /dev/shm/model_output/model-00002-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-run-2-gac-v7/default/model-00002-of-00003.safetensors
Job chaiml-1007-tl-ads-run-2-gac-v7-uploader completed after 73.66s with status: succeeded
Stopping job with name chaiml-1007-tl-ads-run-2-gac-v7-uploader
Pipeline stage VLLMUploader completed in 74.35s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.35s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-1007-tl-ads-run-2-gac-v7
Waiting for inference service chaiml-1007-tl-ads-run-2-gac-v7 to be ready
2026-03-20T23:06:13.189185+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v6
2026-03-20T23:06:15.346832+00:00 monitor updated for chaiml-1007-tl-ads-run-2-gac_v7
Inference service chaiml-1007-tl-ads-aggr-36615-v6 ready after 160.37178206443787s
Pipeline stage VLLMDeployer completed in 160.87s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3519859313964844s
Received healthy response to inference request in 2.303105592727661s
Received healthy response to inference request in 2.3360178470611572s
Received healthy response to inference request in 2.47068190574646s
Received healthy response to inference request in 2.2431447505950928s
Received healthy response to inference request in 2.5983519554138184s
Received healthy response to inference request in 2.2674243450164795s
Received healthy response to inference request in 2.243253231048584s
Received healthy response to inference request in 2.3344945907592773s
Received healthy response to inference request in 2.3123679161071777s
2026-03-20T23:07:13.330923+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v6
Received healthy response to inference request in 2.290968656539917s
2026-03-20T23:07:15.481237+00:00 monitor updated for chaiml-1007-tl-ads-run-2-gac_v7
Received healthy response to inference request in 2.2181999683380127s
Received healthy response to inference request in 2.2375235557556152s
Received healthy response to inference request in 2.355177402496338s
Received healthy response to inference request in 2.27453875541687s
Received healthy response to inference request in 2.277688503265381s
Received healthy response to inference request in 2.235273838043213s
Received healthy response to inference request in 2.6124281883239746s
Received healthy response to inference request in 2.338275909423828s
Received healthy response to inference request in 2.4345953464508057s
Received healthy response to inference request in 2.2363219261169434s
Received healthy response to inference request in 2.228494167327881s
Received healthy response to inference request in 2.3223659992218018s
Received healthy response to inference request in 2.5668790340423584s
Received healthy response to inference request in 2.296765089035034s
Received healthy response to inference request in 3.0181686878204346s
Received healthy response to inference request in 2.269207239151001s
Received healthy response to inference request in 2.242203712463379s
Received healthy response to inference request in 2.3587756156921387s
Received healthy response to inference request in 2.2577638626098633s
30 requests
0 failed requests
5th percentile: 2.23154501914978
10th percentile: 2.2362171173095704
20th percentile: 2.24295654296875
30th percentile: 2.264526200294495
40th percentile: 2.2764286041259765
50th percentile: 2.2999353408813477
60th percentile: 2.327217435836792
70th percentile: 2.342388916015625
80th percentile: 2.3739395618438723
90th percentile: 2.5700263261795047
95th percentile: 2.6060938835144043
99th percentile: 2.9005039429664614
mean time: 2.351081450780233
Pipeline stage StressChecker completed in 75.35s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.80s
Shutdown handler de-registered
chaiml-1007-tl-ads-aggr_36615_v6 status is now deployed due to DeploymentManager action
chaiml-1007-tl-ads-aggr_36615_v6 status is now inactive due to auto deactivation removed underperforming models