Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-1007-tl-ads-with-12871-v8-uploader
Waiting for job on chaiml-1007-tl-ads-with-12871-v8-uploader to finish
chaiml-1007-tl-ads-with-12871-v8-uploader: Using quantization_mode: fp8
Inference service chaiml-1007-tl-ads-loving-v12 ready after 160.42243766784668s
Pipeline stage VLLMDeployer completed in 161.05s
run pipeline stage %s
Running pipeline stage StressChecker
chaiml-1007-tl-ads-with-12871-v8-uploader: Checking if ChaiML/1007-tl-ads-with-openai-filter-FP8 already exists in ChaiML
chaiml-1007-tl-ads-with-12871-v8-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-1007-tl-ads-with-12871-v8-uploader: Downloading snapshot of ChaiML/1007-tl-ads-with-openai-filter-FP8...
chaiml-1007-tl-ads-with-12871-v8-uploader: Downloaded in 7.932s
chaiml-1007-tl-ads-with-12871-v8-uploader: Processed model ChaiML/1007-tl-ads-with-openai-filter in 11.455s
chaiml-1007-tl-ads-with-12871-v8-uploader: creating bucket guanaco-vllm-models
chaiml-1007-tl-ads-with-12871-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v8-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-1007-tl-ads-with-12871-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-1007-tl-ads-with-12871-v8-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-1007-tl-ads-with-12871-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v8-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-with-12871-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v8-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-with-12871-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v8-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-with-12871-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v8-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-with-12871-v8-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-1007-tl-ads-with-12871-v8-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-1007-tl-ads-with-12871-v8-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-1007-tl-ads-with-12871-v8-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-1007-tl-ads-with-12871-v8-uploader: Bucket 's3://guanaco-vllm-models/' created
Received healthy response to inference request in 2.6340601444244385s
chaiml-1007-tl-ads-with-12871-v8-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v8/default
2026-03-20T23:06:11.940195+00:00 monitor updated for chaiml-1007-tl-ads-loving_v12
Received healthy response to inference request in 2.3372130393981934s
Received healthy response to inference request in 2.2169463634490967s
2026-03-20T23:06:16.319830+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v8
Received healthy response to inference request in 2.22516131401062s
chaiml-1007-tl-ads-with-12871-v8-uploader: cp /dev/shm/model_output/model-00003-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v8/default/model-00003-of-00003.safetensors
Received healthy response to inference request in 3.2151036262512207s
Received healthy response to inference request in 2.2451698780059814s
Received healthy response to inference request in 2.276319742202759s
Received healthy response to inference request in 2.2580530643463135s
Received healthy response to inference request in 2.242239475250244s
chaiml-1007-tl-ads-with-12871-v8-uploader: cp /dev/shm/model_output/model-00001-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v8/default/model-00001-of-00003.safetensors
chaiml-1007-tl-ads-with-12871-v8-uploader: cp /dev/shm/model_output/model-00002-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v8/default/model-00002-of-00003.safetensors
Job chaiml-1007-tl-ads-with-12871-v8-uploader completed after 73.4s with status: succeeded
Stopping job with name chaiml-1007-tl-ads-with-12871-v8-uploader
Pipeline stage VLLMUploader completed in 74.06s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.37s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-1007-tl-ads-with-12871-v8
Waiting for inference service chaiml-1007-tl-ads-with-12871-v8 to be ready
Received healthy response to inference request in 2.2487056255340576s
Received healthy response to inference request in 2.291969060897827s
Received healthy response to inference request in 2.317229747772217s
Received healthy response to inference request in 2.2452914714813232s
Received healthy response to inference request in 2.378568172454834s
Received healthy response to inference request in 2.2505688667297363s
Received healthy response to inference request in 2.304598331451416s
Received healthy response to inference request in 2.2544751167297363s
Received healthy response to inference request in 2.311227321624756s
Received healthy response to inference request in 2.234191417694092s
Received healthy response to inference request in 3.698338270187378s
Received healthy response to inference request in 2.232532501220703s
Received healthy response to inference request in 2.5164575576782227s
Received healthy response to inference request in 2.316025972366333s
Received healthy response to inference request in 2.297816038131714s
Received healthy response to inference request in 2.2274880409240723s
Received healthy response to inference request in 2.246910810470581s
2026-03-20T23:07:12.084360+00:00 monitor updated for chaiml-1007-tl-ads-loving_v12
Received healthy response to inference request in 2.254387617111206s
Received healthy response to inference request in 2.2634520530700684s
2026-03-20T23:07:16.642006+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v8
Received healthy response to inference request in 2.2861993312835693s
Received healthy response to inference request in 2.2311487197875977s
30 requests
0 failed requests
5th percentile: 2.2262083411216738
10th percentile: 2.230782651901245
20th percentile: 2.240629863739014
30th percentile: 2.246425008773804
40th percentile: 2.252860116958618
50th percentile: 2.260752558708191
60th percentile: 2.2885072231292725
70th percentile: 2.306587028503418
80th percentile: 2.3212264060974124
90th percentile: 2.5282178163528446
95th percentile: 2.953634059429167
99th percentile: 3.5582002234458927
mean time: 2.3685949563980104
Pipeline stage StressChecker completed in 75.93s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.96s
Shutdown handler de-registered
chaiml-1007-tl-ads-loving_v12 status is now deployed due to DeploymentManager action
2026-03-20T23:08:16.797870+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v8
Inference service chaiml-1007-tl-ads-with-12871-v8 ready after 150.39436650276184s
Pipeline stage VLLMDeployer completed in 151.01s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3069541454315186s
Received healthy response to inference request in 2.673292875289917s
Received healthy response to inference request in 2.317249298095703s
Received healthy response to inference request in 2.3567802906036377s
Received healthy response to inference request in 2.219604253768921s
2026-03-20T23:09:16.889964+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v8
Received healthy response to inference request in 2.307649612426758s
Received healthy response to inference request in 2.2433719635009766s
Received healthy response to inference request in 2.302966356277466s
Received healthy response to inference request in 2.255441188812256s
Received healthy response to inference request in 2.2276766300201416s
Received healthy response to inference request in 2.2453737258911133s
Received healthy response to inference request in 2.2144784927368164s
Received healthy response to inference request in 2.3133037090301514s
Received healthy response to inference request in 2.2507903575897217s
Received healthy response to inference request in 2.233492851257324s
Received healthy response to inference request in 2.292177677154541s
Received healthy response to inference request in 2.2409801483154297s
Received healthy response to inference request in 2.3007593154907227s
Received healthy response to inference request in 2.2312419414520264s
Received healthy response to inference request in 2.2343387603759766s
Received healthy response to inference request in 2.235426425933838s
Received healthy response to inference request in 2.333238124847412s
Received healthy response to inference request in 2.2507901191711426s
Received healthy response to inference request in 2.239034414291382s
Received healthy response to inference request in 2.2132105827331543s
Received healthy response to inference request in 2.213552713394165s
Received healthy response to inference request in 2.234868288040161s
Received healthy response to inference request in 2.2525172233581543s
Received healthy response to inference request in 2.2824912071228027s
Received healthy response to inference request in 2.2345898151397705s
30 requests
0 failed requests
5th percentile: 2.213969314098358
10th percentile: 2.2190916776657104
20th percentile: 2.233042669296265
30th percentile: 2.234784746170044
40th percentile: 2.2402018547058105
50th percentile: 2.248081922531128
60th percentile: 2.253686809539795
70th percentile: 2.2947521686553953
80th percentile: 2.3070932388305665
90th percentile: 2.318848180770874
95th percentile: 2.3461863160133363
99th percentile: 2.5815042257308964
mean time: 2.27525475025177
Pipeline stage StressChecker completed in 70.86s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.61s
Shutdown handler de-registered
chaiml-1007-tl-ads-with_12871_v8 status is now deployed due to DeploymentManager action
chaiml-1007-tl-ads-with_12871_v8 status is now inactive due to auto deactivation removed underperforming models