Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-bobo-creative-lr-68876-v1-uploader
Waiting for job on chaiml-bobo-creative-lr-68876-v1-uploader to finish
google-gemma-4-31b-it-v28-uploader: Using quantization_mode: none
google-gemma-4-31b-it-v28-uploader: Downloading snapshot of google/gemma-4-31B-it...
chaiml-bobo-creative-lr2-5224-v1-uploader: Using quantization_mode: none
chaiml-bobo-creative-lr2-5224-v1-uploader: Downloading snapshot of ChaiML/bobo_creative_lr2_full_run_fixed-step2806-merged...
chaiml-bobo-creative-lr-68876-v1-uploader: Using quantization_mode: none
chaiml-bobo-creative-lr-68876-v1-uploader: Downloading snapshot of ChaiML/bobo_creative_lr1_full_run_fixed-step2806-merged...
google-gemma-4-31b-it-v28-uploader: Downloaded in 33.166s
chaiml-bobo-creative-lr2-5224-v1-uploader: Downloaded in 25.341s
2026-04-08T16:00:46.750807+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:00:51.444362+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
google-gemma-4-31b-it-v28-uploader: Processed model google/gemma-4-31B-it in 55.224s
google-gemma-4-31b-it-v28-uploader: creating bucket guanaco-vllm-models
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
google-gemma-4-31b-it-v28-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: if re.search("-\.", bucket, re.UNICODE):
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: if re.search("\.\.", bucket, re.UNICODE):
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
google-gemma-4-31b-it-v28-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
google-gemma-4-31b-it-v28-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bobo-creative-lr2-5224-v1-uploader: Processed model ChaiML/bobo_creative_lr2_full_run_fixed-step2806-merged in 48.750s
google-gemma-4-31b-it-v28-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bobo-creative-lr2-5224-v1-uploader: creating bucket guanaco-vllm-models
google-gemma-4-31b-it-v28-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default
2026-04-08T16:00:56.858964+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/.gitattributes
chaiml-bobo-creative-lr2-5224-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/model.safetensors.index.json
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/processor_config.json
chaiml-bobo-creative-lr2-5224-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/config.json
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/generation_config.json
chaiml-bobo-creative-lr2-5224-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/tokenizer_config.json
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/chat_template.jinja
chaiml-bobo-creative-lr2-5224-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/README.md
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/tokenizer.json
chaiml-bobo-creative-lr2-5224-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-bobo-creative-lr2-5224-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-bobo-creative-lr2-5224-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bobo-creative-lr2-5224-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bobo-creative-lr-68876-v1-uploader: Processed model ChaiML/bobo_creative_lr1_full_run_fixed-step2806-merged in 48.220s
chaiml-bobo-creative-lr2-5224-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default
chaiml-bobo-creative-lr-68876-v1-uploader: creating bucket guanaco-vllm-models
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/processor_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/config.json
chaiml-bobo-creative-lr-68876-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/.gitattributes
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/README.md
chaiml-bobo-creative-lr-68876-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/generation_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/args.json
chaiml-bobo-creative-lr-68876-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/chat_template.jinja
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/tokenizer_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model.safetensors.index.json
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/tokenizer.json
chaiml-bobo-creative-lr-68876-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr-68876-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-bobo-creative-lr-68876-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-bobo-creative-lr-68876-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bobo-creative-lr-68876-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bobo-creative-lr-68876-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/README.md
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/args.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/config.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/.gitattributes
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/generation_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/chat_template.jinja
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/tokenizer_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/processor_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/model.safetensors.index.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/tokenizer.json
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00004-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00004-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00007-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00007-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00009-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00009-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00011-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00011-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00008-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00008-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00010-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00010-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00012-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00012-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00003-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00003-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00013-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00013-of-00013.safetensors
Job chaiml-bobo-creative-lr2-5224-v1-uploader completed after 87.13s with status: succeeded
Stopping job with name chaiml-bobo-creative-lr2-5224-v1-uploader
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/model-00002-of-00002.safetensors
Pipeline stage VLLMUploader completed in 89.13s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.81s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Job chaiml-bobo-creative-lr-68876-v1-uploader completed after 84.69s with status: succeeded
Stopping job with name chaiml-bobo-creative-lr-68876-v1-uploader
Pipeline stage VLLMUploader completed in 86.88s
run pipeline stage %s
Pipeline stage VLLMTemplater completed in 1.70s
Running pipeline stage VLLMUploaderAMD
run pipeline stage %s
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploaderAMD completed in 0.79s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.50s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-bobo-creative-lr-68876-v1
Waiting for inference service chaiml-bobo-creative-lr-68876-v1 to be ready
Creating inference service chaiml-bobo-creative-lr2-5224-v1
Waiting for inference service chaiml-bobo-creative-lr2-5224-v1 to be ready
2026-04-08T16:01:47.200024+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:01:51.804323+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:01:57.573444+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/model-00001-of-00002.safetensors
Job google-gemma-4-31b-it-v28-uploader completed after 166.41s with status: succeeded
Stopping job with name google-gemma-4-31b-it-v28-uploader
Pipeline stage VLLMUploader completed in 168.21s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.73s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.42s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service google-gemma-4-31b-it-v28
Waiting for inference service google-gemma-4-31b-it-v28 to be ready
2026-04-08T16:02:47.684788+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:02:52.245661+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:02:58.112247+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
2026-04-08T16:03:48.213326+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:03:52.765656+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:03:58.620569+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
2026-04-08T16:04:48.722447+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:04:53.239044+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:04:59.131917+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
2026-04-08T16:05:49.276194+00:00 monitor updated for google-gemma-4-31b-it_v28
Inference service chaiml-bobo-creative-lr-68876-v1 ready after 253.46247673034668s
Pipeline stage VLLMDeployer completed in 260.80s
run pipeline stage %s
Running pipeline stage StressChecker
2026-04-08T16:05:53.699851+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:05:59.620062+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
Received healthy response to inference request in 12.63709282875061s
Received healthy response to inference request in 4.882525682449341s
Received healthy response to inference request in 4.185794353485107s
Received healthy response to inference request in 4.281155824661255s
Received healthy response to inference request in 4.371833086013794s
Received healthy response to inference request in 4.2042131423950195s
Received healthy response to inference request in 4.204666614532471s
Received healthy response to inference request in 4.858266115188599s
Received healthy response to inference request in 4.154895782470703s
Received healthy response to inference request in 4.26915979385376s
2026-04-08T16:06:50.081572+00:00 monitor updated for google-gemma-4-31b-it_v28
Received healthy response to inference request in 4.471526145935059s
2026-04-08T16:06:54.175842+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
Received healthy response to inference request in 4.175513982772827s
2026-04-08T16:07:00.150301+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
Received healthy response to inference request in 4.217221260070801s
Received healthy response to inference request in 4.117532730102539s
Received healthy response to inference request in 4.521929502487183s
Received healthy response to inference request in 4.207247972488403s
Inference service chaiml-bobo-creative-lr2-5224-v1 ready after 334.358252286911s
Pipeline stage VLLMDeployer completed in 353.68s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.330422639846802s
Received healthy response to inference request in 4.4335198402404785s
Received healthy response to inference request in 11.503621339797974s
Received healthy response to inference request in 4.813175916671753s
Received healthy response to inference request in 11.63954472541809s
Received healthy response to inference request in 4.150347471237183s
Received healthy response to inference request in 4.197232246398926s
Received healthy response to inference request in 4.221250295639038s
Received healthy response to inference request in 4.400445222854614s
2026-04-08T16:07:50.581239+00:00 monitor updated for google-gemma-4-31b-it_v28
Received healthy response to inference request in 4.177430152893066s
2026-04-08T16:07:54.694589+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
Received healthy response to inference request in 4.152604341506958s
Received healthy response to inference request in 11.8175950050354s
2026-04-08T16:08:00.677361+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
Received healthy response to inference request in 4.1863086223602295s
Received healthy response to inference request in 4.669449090957642s
Received healthy response to inference request in 11.591830968856812s
Received healthy response to inference request in 4.439723968505859s
Received healthy response to inference request in 4.1950483322143555s
Inference service google-gemma-4-31b-it-v28 ready after 335.13281655311584s
Pipeline stage VLLMDeployer completed in 337.70s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.1240599155426025s
Received healthy response to inference request in 11.994389295578003s
Received healthy response to inference request in 4.1125993728637695s
Received healthy response to inference request in 4.3610169887542725s
Received healthy response to inference request in 4.140762805938721s
Received healthy response to inference request in 11.720366954803467s
Received healthy response to inference request in 4.248918533325195s
Received healthy response to inference request in 4.150777101516724s
Received healthy response to inference request in 4.208745002746582s
30 requests
Received healthy response to inference request in 4.112037181854248s
0 failed requests
5th percentile: 4.153635489940643
10th percentile: 4.173452162742615
20th percentile: 4.186205768585205
30th percentile: 4.204530572891235
40th percentile: 4.213830757141113
50th percentile: 4.275157809257507
60th percentile: 4.365343427658081
Received healthy response to inference request in 11.668477773666382s
70th percentile: 4.435381078720093
80th percentile: 4.551433420181275
90th percentile: 5.558227586746225
Received healthy response to inference request in 4.813737630844116s
95th percentile: 11.834709239006042
99th percentile: 12.450708804130555
mean time: 5.106677516301473
Pipeline stage StressChecker completed in 175.04s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 4.1610023975372314s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.70s
Shutdown handler de-registered
chaiml-bobo-creative-lr_68876_v1 status is now deployed due to DeploymentManager action
chaiml-bobo-creative-lr_68876_v1 status is now inactive due to auto deactivation removed underperforming models