Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gemma4-bobo-crea-55391-v1-uploader
Waiting for job on chaiml-gemma4-bobo-crea-55391-v1-uploader to finish
chaiml-gemma4-bobo-crea-55391-v1-uploader: Using quantization_mode: none
chaiml-gemma4-bobo-crea-55391-v1-uploader: Downloading snapshot of ChaiML/gemma4_bobo_creative_fixed_full-step2806-merged...
chaiml-gemma4-bobo-crea-55391-v1-uploader: Downloaded in 25.869s
2026-04-07T06:44:20.583600+00:00 monitor updated for chaiml-gemma4-bobo-crea_55391_v1
chaiml-gemma4-bobo-crea-55391-v1-uploader: Processed model ChaiML/gemma4_bobo_creative_fixed_full-step2806-merged in 49.626s
chaiml-gemma4-bobo-crea-55391-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gemma4-bobo-crea-55391-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gemma4-bobo-crea-55391-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gemma4-bobo-crea-55391-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gemma4-bobo-crea-55391-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gemma4-bobo-crea-55391-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gemma4-bobo-crea-55391-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gemma4-bobo-crea-55391-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gemma4-bobo-crea-55391-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gemma4-bobo-crea-55391-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gemma4-bobo-crea-55391-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-gemma4-bobo-crea-55391-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gemma4-bobo-crea-55391-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-gemma4-bobo-crea-55391-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gemma4-bobo-crea-55391-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gemma4-bobo-crea-55391-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gemma4-bobo-crea-55391-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gemma4-bobo-crea-55391-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gemma4-bobo-crea-55391-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/config.json
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/README.md
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/processor_config.json
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/args.json
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/generation_config.json
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/tokenizer_config.json
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/.gitattributes
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model.safetensors.index.json
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/chat_template.jinja
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/tokenizer.json
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00013-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00013-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00010-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00010-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00011-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00011-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00005-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00005-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00009-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00009-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00004-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00004-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00012-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00012-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00007-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00007-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00008-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00008-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00003-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00003-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00001-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00001-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00002-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00002-of-00013.safetensors
chaiml-gemma4-bobo-crea-55391-v1-uploader: cp /dev/shm/model_output/model-00006-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gemma4-bobo-crea-55391-v1/default/model-00006-of-00013.safetensors
Job chaiml-gemma4-bobo-crea-55391-v1-uploader completed after 77.47s with status: succeeded
Stopping job with name chaiml-gemma4-bobo-crea-55391-v1-uploader
Pipeline stage VLLMUploader completed in 78.61s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.22s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.21s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gemma4-bobo-crea-55391-v1
Waiting for inference service chaiml-gemma4-bobo-crea-55391-v1 to be ready
2026-04-07T06:45:21.019440+00:00 monitor updated for chaiml-gemma4-bobo-crea_55391_v1
2026-04-07T06:46:21.245668+00:00 monitor updated for chaiml-gemma4-bobo-crea_55391_v1
2026-04-07T06:47:21.436023+00:00 monitor updated for chaiml-gemma4-bobo-crea_55391_v1
2026-04-07T06:48:21.671658+00:00 monitor updated for chaiml-gemma4-bobo-crea_55391_v1
Inference service chaiml-gemma4-bobo-crea-55391-v1 ready after 263.5630724430084s
Pipeline stage VLLMDeployer completed in 264.68s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 7.555624723434448s
Received healthy response to inference request in 2.7374119758605957s
Received healthy response to inference request in 2.7468626499176025s
2026-04-07T06:49:21.858489+00:00 monitor updated for chaiml-gemma4-bobo-crea_55391_v1
Received healthy response to inference request in 2.658223867416382s
Received healthy response to inference request in 2.5798966884613037s
Received healthy response to inference request in 2.7356319427490234s
Received healthy response to inference request in 2.65876841545105s
Received healthy response to inference request in 2.6358630657196045s
Received healthy response to inference request in 2.6752281188964844s
Received healthy response to inference request in 2.751558303833008s
Received healthy response to inference request in 2.81807804107666s
Received healthy response to inference request in 2.6072864532470703s
Received healthy response to inference request in 2.743481397628784s
Received healthy response to inference request in 2.5645148754119873s
Received healthy response to inference request in 7.8925940990448s
Received healthy response to inference request in 2.959909439086914s
Received healthy response to inference request in 2.5776820182800293s
Received healthy response to inference request in 2.8463878631591797s
Received healthy response to inference request in 2.70577335357666s
Received healthy response to inference request in 2.848705530166626s
Received healthy response to inference request in 2.7695350646972656s
Received healthy response to inference request in 2.7903401851654053s
2026-04-07T06:50:22.042546+00:00 monitor updated for chaiml-gemma4-bobo-crea_55391_v1
Received healthy response to inference request in 2.8191773891448975s
Received healthy response to inference request in 2.7508702278137207s
Received healthy response to inference request in 2.768806219100952s
Received healthy response to inference request in 2.80995512008667s
Received healthy response to inference request in 2.943347454071045s
Received healthy response to inference request in 2.8066959381103516s
Received healthy response to inference request in 2.6250641345977783s
Received healthy response to inference request in 3.091022491455078s
30 requests
0 failed requests
5th percentile: 2.578678619861603
10th percentile: 2.6045474767684937
20th percentile: 2.6537517070770265
30th percentile: 2.6966097831726072
40th percentile: 2.7410536289215086
50th percentile: 2.7512142658233643
60th percentile: 2.7778571128845213
70th percentile: 2.812391996383667
80th percentile: 2.8468513965606688
90th percentile: 2.9730207443237306
95th percentile: 5.546553719043719
99th percentile: 7.794872980117798
mean time: 3.082476568222046
Pipeline stage StressChecker completed in 98.85s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.34s
Shutdown handler de-registered
chaiml-gemma4-bobo-crea_55391_v1 status is now deployed due to DeploymentManager action
chaiml-gemma4-bobo-crea_55391_v1 status is now inactive due to auto deactivation removed underperforming models