Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-mega-v1-winopus-97704-v1-uploader
Waiting for job on chaiml-mega-v1-winopus-97704-v1-uploader to finish
chaiml-mega-v1-winopus-97704-v1-uploader: Using quantization_mode: none
chaiml-mega-v1-winopus-97704-v1-uploader: Downloading snapshot of ChaiML/mega-v1-winopus-q35b-lr5e6ep2g8...
2026-03-23T15:18:52.548673+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
chaiml-mega-v1-winopus-97704-v1-uploader: Downloaded in 46.755s
chaiml-mega-v1-winopus-97704-v1-uploader: Processed model ChaiML/mega-v1-winopus-q35b-lr5e6ep2g8 in 72.276s
chaiml-mega-v1-winopus-97704-v1-uploader: creating bucket guanaco-vllm-models
chaiml-mega-v1-winopus-97704-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-winopus-97704-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-mega-v1-winopus-97704-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-mega-v1-winopus-97704-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-mega-v1-winopus-97704-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-winopus-97704-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-mega-v1-winopus-97704-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-winopus-97704-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-mega-v1-winopus-97704-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-winopus-97704-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-mega-v1-winopus-97704-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-winopus-97704-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-mega-v1-winopus-97704-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-mega-v1-winopus-97704-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-mega-v1-winopus-97704-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-mega-v1-winopus-97704-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-mega-v1-winopus-97704-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-mega-v1-winopus-97704-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/.gitattributes
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/generation_config.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/args.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/config.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/chat_template.jinja
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/special_tokens_map.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/processor_config.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/tokenizer_config.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/training_args.bin
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/trainer_state.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/trainer_state.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/README.md
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/added_tokens.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/preprocessor_config.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/merges.txt
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/model.safetensors.index.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/vocab.json
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/tokenizer.json
2026-03-23T15:19:52.781295+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/model-00002-of-00002.safetensors
2026-03-23T15:20:52.946529+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
Failed to get request counts for guanaco-submitter. Falling back to default
2026-03-23T15:21:53.077299+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
chaiml-mega-v1-winopus-97704-v1-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-mega-v1-winopus-97704-v1/default/model-00001-of-00002.safetensors
Job chaiml-mega-v1-winopus-97704-v1-uploader completed after 285.49s with status: succeeded
Stopping job with name chaiml-mega-v1-winopus-97704-v1-uploader
Pipeline stage VLLMUploader completed in 286.05s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.35s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-mega-v1-winopus-97704-v1
Waiting for inference service chaiml-mega-v1-winopus-97704-v1 to be ready
2026-03-23T15:22:53.176186+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
2026-03-23T15:23:53.275978+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
2026-03-23T15:24:53.374751+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
2026-03-23T15:25:53.502082+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
Inference service chaiml-mega-v1-winopus-97704-v1 ready after 240.73563504219055s
Pipeline stage VLLMDeployer completed in 241.62s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-23T15:26:53.635499+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-23T15:27:53.759046+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.9147257804870605s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-23T15:28:53.853012+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.0462400913238525s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 6.7385852336883545s
2026-03-23T15:29:53.951109+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.3997724056243896s
Received healthy response to inference request in 2.3772330284118652s
Received healthy response to inference request in 1.2833120822906494s
Received healthy response to inference request in 2.319023847579956s
Received healthy response to inference request in 1.1110076904296875s
Received healthy response to inference request in 1.2341012954711914s
Received healthy response to inference request in 13.151174068450928s
Received healthy response to inference request in 1.1189825534820557s
Received healthy response to inference request in 1.1218833923339844s
Received healthy response to inference request in 1.2242717742919922s
Received healthy response to inference request in 1.2205960750579834s
Received healthy response to inference request in 1.3091323375701904s
Received healthy response to inference request in 2.0059151649475098s
Received healthy response to inference request in 2.2738382816314697s
Received healthy response to inference request in 1.1275691986083984s
Received healthy response to inference request in 1.2291252613067627s
Received healthy response to inference request in 1.123065710067749s
Received healthy response to inference request in 1.1646904945373535s
30 requests
9 failed requests
5th percentile: 1.1202879309654237
10th percentile: 1.1229474782943725
20th percentile: 1.2094149589538574
30th percentile: 1.2326084852218628
40th percentile: 1.36351637840271
50th percentile: 2.296431064605713
60th percentile: 4.393634366989135
70th percentile: 15.234784626960735
80th percentile: 20.122223234176637
90th percentile: 20.148075652122497
95th percentile: 20.179009246826173
99th percentile: 20.344372663497925
mean time: 7.8333101034164425
%s, retrying in %s seconds...
Received healthy response to inference request in 1.0147056579589844s
Received healthy response to inference request in 1.170839548110962s
Received healthy response to inference request in 1.3065528869628906s
Received healthy response to inference request in 1.0873281955718994s
Received healthy response to inference request in 1.2375242710113525s
Received healthy response to inference request in 1.3083744049072266s
Received healthy response to inference request in 1.1610794067382812s
Received healthy response to inference request in 1.2365179061889648s
Received healthy response to inference request in 1.1537625789642334s
Received healthy response to inference request in 1.1659369468688965s
Received healthy response to inference request in 1.205249309539795s
2026-03-23T15:30:54.051712+00:00 monitor updated for chaiml-mega-v1-winopus-_97704_v1
Received healthy response to inference request in 1.261077642440796s
Received healthy response to inference request in 1.1729378700256348s
Received healthy response to inference request in 1.096606731414795s
Received healthy response to inference request in 1.1580615043640137s
Received healthy response to inference request in 1.126603126525879s
Received healthy response to inference request in 1.1645851135253906s
Received healthy response to inference request in 2.0894763469696045s
Received healthy response to inference request in 1.30755615234375s
Received healthy response to inference request in 1.208496332168579s
Received healthy response to inference request in 1.2232110500335693s
Received healthy response to inference request in 1.2405569553375244s
Received healthy response to inference request in 1.1475565433502197s
Received healthy response to inference request in 1.3731193542480469s
Received healthy response to inference request in 1.1662676334381104s
Received healthy response to inference request in 1.2083442211151123s
Received healthy response to inference request in 1.1843657493591309s
Received healthy response to inference request in 1.094531536102295s
Received healthy response to inference request in 1.1875813007354736s
Received healthy response to inference request in 1.095463752746582s
30 requests
0 failed requests
5th percentile: 1.0905696988105773
10th percentile: 1.0953705310821533
20th percentile: 1.1433658599853516
30th percentile: 1.1601740360260009
40th percentile: 1.1661353588104248
50th percentile: 1.1786518096923828
60th percentile: 1.206487274169922
70th percentile: 1.2272031068801879
80th percentile: 1.2446610927581787
90th percentile: 1.3076379776000977
95th percentile: 1.3439841270446775
99th percentile: 1.8817328190803533
mean time: 1.2184756676355997
Pipeline stage StressChecker completed in 276.78s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.37s
Shutdown handler de-registered
chaiml-mega-v1-winopus-_97704_v1 status is now deployed due to DeploymentManager action
chaiml-mega-v1-winopus-_97704_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-mega-v1-winopus-_97704_v1 status is now torndown due to DeploymentManager action