Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-mega-v1-plc-q35b-52016-v1-uploader
Waiting for job on chaiml-mega-v1-plc-q35b-52016-v1-uploader to finish
chaiml-mega-v1-plc-q35b-52016-v1-uploader: Using quantization_mode: none
chaiml-mega-v1-plc-q35b-52016-v1-uploader: Downloading snapshot of CHaiML/mega-v1-plc-q35b-lr5e6ep2g8...
chaiml-mega-v1-plc-q35b-52016-v1-uploader: Downloaded in 40.227s
2026-03-25T03:48:49.605762+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
chaiml-mega-v1-plc-q35b-52016-v1-uploader: Processed model CHaiML/mega-v1-plc-q35b-lr5e6ep2g8 in 66.203s
chaiml-mega-v1-plc-q35b-52016-v1-uploader: creating bucket guanaco-vllm-models
chaiml-mega-v1-plc-q35b-52016-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-plc-q35b-52016-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-mega-v1-plc-q35b-52016-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-mega-v1-plc-q35b-52016-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-mega-v1-plc-q35b-52016-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-plc-q35b-52016-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-mega-v1-plc-q35b-52016-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-plc-q35b-52016-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-mega-v1-plc-q35b-52016-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-plc-q35b-52016-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-mega-v1-plc-q35b-52016-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-plc-q35b-52016-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-mega-v1-plc-q35b-52016-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-mega-v1-plc-q35b-52016-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-mega-v1-plc-q35b-52016-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-mega-v1-plc-q35b-52016-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-mega-v1-plc-q35b-52016-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-mega-v1-plc-q35b-52016-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/README.md
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/.gitattributes
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/added_tokens.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/config.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/args.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/preprocessor_config.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/trainer_state.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/trainer_state.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/special_tokens_map.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/model.safetensors.index.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/generation_config.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/merges.txt
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/chat_template.jinja
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/tokenizer_config.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/training_args.bin
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/processor_config.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/vocab.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/tokenizer.json
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/model-00002-of-00002.safetensors
2026-03-25T03:49:49.692575+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
chaiml-mega-v1-plc-q35b-52016-v1-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-mega-v1-plc-q35b-52016-v1/default/model-00001-of-00002.safetensors
Job chaiml-mega-v1-plc-q35b-52016-v1-uploader completed after 163.47s with status: succeeded
Stopping job with name chaiml-mega-v1-plc-q35b-52016-v1-uploader
Pipeline stage VLLMUploader completed in 163.98s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.08s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-mega-v1-plc-q35b-52016-v1
Waiting for inference service chaiml-mega-v1-plc-q35b-52016-v1 to be ready
2026-03-25T03:50:49.785092+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
2026-03-25T03:51:49.872695+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
2026-03-25T03:52:49.965928+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
Inference service chaiml-mega-v1-plc-q35b-52016-v1 ready after 190.74777364730835s
Pipeline stage VLLMDeployer completed in 191.26s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-25T03:53:50.055956+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T03:54:50.141503+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.701648712158203s
Received healthy response to inference request in 2.3708834648132324s
Received healthy response to inference request in 1.260066270828247s
Received healthy response to inference request in 1.7275629043579102s
Received healthy response to inference request in 1.6451570987701416s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T03:55:50.226829+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
Received healthy response to inference request in 4.014159679412842s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.378009557723999s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 6.883640766143799s
Received healthy response to inference request in 1.1203036308288574s
Received healthy response to inference request in 1.1152539253234863s
Received healthy response to inference request in 1.6757230758666992s
Received healthy response to inference request in 1.3566381931304932s
Received healthy response to inference request in 1.8676574230194092s
2026-03-25T03:56:50.312968+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
Received healthy response to inference request in 1.274376630783081s
Received healthy response to inference request in 1.9459228515625s
Received healthy response to inference request in 1.2020576000213623s
Received healthy response to inference request in 11.148558616638184s
Received healthy response to inference request in 1.2719848155975342s
Received healthy response to inference request in 1.3190793991088867s
Received healthy response to inference request in 1.7808640003204346s
Received healthy response to inference request in 1.848829746246338s
Received healthy response to inference request in 2.011077404022217s
Received healthy response to inference request in 1.2230377197265625s
30 requests
7 failed requests
5th percentile: 1.1570929169654847
10th percentile: 1.2209397077560424
20th percentile: 1.2738982677459716
30th percentile: 1.3715981483459472
40th percentile: 1.7068269729614258
50th percentile: 1.8582435846328735
60th percentile: 2.1549998283386227
70th percentile: 4.875004005432121
80th percentile: 20.104313468933107
90th percentile: 20.112022113800048
95th percentile: 20.122648656368256
99th percentile: 20.130653722286226
mean time: 6.531831518809001
%s, retrying in %s seconds...
Received healthy response to inference request in 1.1083388328552246s
Received healthy response to inference request in 1.1260361671447754s
Received healthy response to inference request in 1.2092921733856201s
Received healthy response to inference request in 1.0869526863098145s
Received healthy response to inference request in 1.8304626941680908s
Received healthy response to inference request in 1.2340295314788818s
Received healthy response to inference request in 1.1233234405517578s
Received healthy response to inference request in 1.4551057815551758s
Received healthy response to inference request in 1.0397837162017822s
Received healthy response to inference request in 1.3293156623840332s
Received healthy response to inference request in 1.1061501502990723s
Received healthy response to inference request in 1.2308595180511475s
Received healthy response to inference request in 1.232177495956421s
Received healthy response to inference request in 1.5539453029632568s
Received healthy response to inference request in 1.584324598312378s
Received healthy response to inference request in 1.7540333271026611s
Received healthy response to inference request in 1.4225075244903564s
Received healthy response to inference request in 1.2488157749176025s
Received healthy response to inference request in 1.1671438217163086s
Received healthy response to inference request in 1.2291572093963623s
Received healthy response to inference request in 1.3080732822418213s
Received healthy response to inference request in 1.114267110824585s
Received healthy response to inference request in 1.729201316833496s
Received healthy response to inference request in 1.228386640548706s
Received healthy response to inference request in 1.1648569107055664s
2026-03-25T03:57:50.409312+00:00 monitor updated for chaiml-mega-v1-plc-q35b_52016_v1
Received healthy response to inference request in 1.2770934104919434s
Received healthy response to inference request in 1.093601942062378s
Received healthy response to inference request in 1.1524789333343506s
Received healthy response to inference request in 1.1310830116271973s
Received healthy response to inference request in 1.149414300918579s
30 requests
0 failed requests
5th percentile: 1.089944851398468
10th percentile: 1.104895329475403
20th percentile: 1.1215121746063232
30th percentile: 1.1439149141311646
40th percentile: 1.1662290573120118
50th percentile: 1.2287719249725342
60th percentile: 1.2329183101654053
70th percentile: 1.2863873720169066
80th percentile: 1.4290271759033204
90th percentile: 1.59881227016449
95th percentile: 1.7428589224815367
99th percentile: 1.8082981777191163
mean time: 1.2806737422943115
Pipeline stage StressChecker completed in 247.98s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.61s
Shutdown handler de-registered
chaiml-mega-v1-plc-q35b_52016_v1 status is now deployed due to DeploymentManager action
chaiml-mega-v1-plc-q35b_52016_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-mega-v1-plc-q35b_52016_v1 status is now torndown due to DeploymentManager action