google-gemma-4-31b-it

developer_uid: rirv938
submission_id: google-gemma-4-31b-it_v28
model_name: google-gemma-4-31b-it_v28
model_group: google/gemma-4-31B-it
status: inactive
timestamp: 2026-04-08T17:39:58+00:00
num_battles: 10726
num_wins: 5631
celo_rating: 1318.79
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: google/gemma-4-31B-it
model_architecture: Gemma4ForConditionalGeneration
model_num_parameters: 28830461184.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: google-gemma-4-31b-it_v28
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: google/gemma-4-31B-it
model_size: 29B
ranking_group: single
us_pacific_date: 2026-04-08
win_ratio: 0.5249860152899497
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 64, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_start|>', '<turn|>', '###', 'You:', '<|turn>', '<|redacted_im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|turn>system\nYou are "{bot_name}". Stay in character and respond only as "{bot_name}". Create an entertaining dialogue<turn|>\n', 'prompt_template': '', 'bot_template': '<|turn>model\n{message}<turn|>\n', 'user_template': '<|turn>user\n{message}<turn|>\n', 'response_template': '<|turn>model\n<|channel>thought\n<channel|>', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name google-gemma-4-31b-it-v28-uploader
Waiting for job on google-gemma-4-31b-it-v28-uploader to finish
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-bobo-creative-lr2-5224-v1-uploader
Waiting for job on chaiml-bobo-creative-lr2-5224-v1-uploader to finish
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-bobo-creative-lr-68876-v1-uploader
Waiting for job on chaiml-bobo-creative-lr-68876-v1-uploader to finish
google-gemma-4-31b-it-v28-uploader: Using quantization_mode: none
google-gemma-4-31b-it-v28-uploader: Downloading snapshot of google/gemma-4-31B-it...
chaiml-bobo-creative-lr2-5224-v1-uploader: Using quantization_mode: none
chaiml-bobo-creative-lr2-5224-v1-uploader: Downloading snapshot of ChaiML/bobo_creative_lr2_full_run_fixed-step2806-merged...
chaiml-bobo-creative-lr-68876-v1-uploader: Using quantization_mode: none
chaiml-bobo-creative-lr-68876-v1-uploader: Downloading snapshot of ChaiML/bobo_creative_lr1_full_run_fixed-step2806-merged...
google-gemma-4-31b-it-v28-uploader: Downloaded in 33.166s
chaiml-bobo-creative-lr2-5224-v1-uploader: Downloaded in 25.341s
2026-04-08T16:00:46.750807+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:00:51.444362+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
google-gemma-4-31b-it-v28-uploader: Processed model google/gemma-4-31B-it in 55.224s
google-gemma-4-31b-it-v28-uploader: creating bucket guanaco-vllm-models
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
google-gemma-4-31b-it-v28-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader:   if re.search("-\.", bucket, re.UNICODE):
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader:   if re.search("\.\.", bucket, re.UNICODE):
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
google-gemma-4-31b-it-v28-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
google-gemma-4-31b-it-v28-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
google-gemma-4-31b-it-v28-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bobo-creative-lr2-5224-v1-uploader: Processed model ChaiML/bobo_creative_lr2_full_run_fixed-step2806-merged in 48.750s
google-gemma-4-31b-it-v28-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bobo-creative-lr2-5224-v1-uploader: creating bucket guanaco-vllm-models
google-gemma-4-31b-it-v28-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default
2026-04-08T16:00:56.858964+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/.gitattributes
chaiml-bobo-creative-lr2-5224-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/model.safetensors.index.json
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/processor_config.json
chaiml-bobo-creative-lr2-5224-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/config.json
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/generation_config.json
chaiml-bobo-creative-lr2-5224-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/tokenizer_config.json
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/chat_template.jinja
chaiml-bobo-creative-lr2-5224-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/README.md
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/tokenizer.json
chaiml-bobo-creative-lr2-5224-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-bobo-creative-lr2-5224-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-bobo-creative-lr2-5224-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-bobo-creative-lr2-5224-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bobo-creative-lr2-5224-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bobo-creative-lr-68876-v1-uploader: Processed model ChaiML/bobo_creative_lr1_full_run_fixed-step2806-merged in 48.220s
chaiml-bobo-creative-lr2-5224-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default
chaiml-bobo-creative-lr-68876-v1-uploader: creating bucket guanaco-vllm-models
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/processor_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/config.json
chaiml-bobo-creative-lr-68876-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/.gitattributes
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/README.md
chaiml-bobo-creative-lr-68876-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/generation_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/args.json
chaiml-bobo-creative-lr-68876-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/chat_template.jinja
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/tokenizer_config.json
chaiml-bobo-creative-lr-68876-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model.safetensors.index.json
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/tokenizer.json
chaiml-bobo-creative-lr-68876-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr-68876-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-bobo-creative-lr-68876-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-bobo-creative-lr-68876-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-bobo-creative-lr-68876-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bobo-creative-lr-68876-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bobo-creative-lr-68876-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/README.md
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/args.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/config.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/.gitattributes
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/generation_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/chat_template.jinja
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/tokenizer_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/processor_config.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/model.safetensors.index.json
chaiml-bobo-creative-lr-68876-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-68876-v1/default/tokenizer.json
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00004-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00004-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00007-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00007-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00009-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00009-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00011-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00011-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00008-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00008-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00010-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00010-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00012-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00012-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00003-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00003-of-00013.safetensors
chaiml-bobo-creative-lr2-5224-v1-uploader: cp /dev/shm/model_output/model-00013-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr2-5224-v1/default/model-00013-of-00013.safetensors
Job chaiml-bobo-creative-lr2-5224-v1-uploader completed after 87.13s with status: succeeded
Stopping job with name chaiml-bobo-creative-lr2-5224-v1-uploader
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/model-00002-of-00002.safetensors
Pipeline stage VLLMUploader completed in 89.13s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.81s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Job chaiml-bobo-creative-lr-68876-v1-uploader completed after 84.69s with status: succeeded
Stopping job with name chaiml-bobo-creative-lr-68876-v1-uploader
Pipeline stage VLLMUploader completed in 86.88s
run pipeline stage %s
Pipeline stage VLLMTemplater completed in 1.70s
Running pipeline stage VLLMUploaderAMD
run pipeline stage %s
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploaderAMD completed in 0.79s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.50s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-bobo-creative-lr-68876-v1
Waiting for inference service chaiml-bobo-creative-lr-68876-v1 to be ready
Creating inference service chaiml-bobo-creative-lr2-5224-v1
Waiting for inference service chaiml-bobo-creative-lr2-5224-v1 to be ready
2026-04-08T16:01:47.200024+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:01:51.804323+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:01:57.573444+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
google-gemma-4-31b-it-v28-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/google-gemma-4-31b-it-v28/default/model-00001-of-00002.safetensors
Job google-gemma-4-31b-it-v28-uploader completed after 166.41s with status: succeeded
Stopping job with name google-gemma-4-31b-it-v28-uploader
Pipeline stage VLLMUploader completed in 168.21s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.73s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.42s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service google-gemma-4-31b-it-v28
Waiting for inference service google-gemma-4-31b-it-v28 to be ready
2026-04-08T16:02:47.684788+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:02:52.245661+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:02:58.112247+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
2026-04-08T16:03:48.213326+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:03:52.765656+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:03:58.620569+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
2026-04-08T16:04:48.722447+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:04:53.239044+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:04:59.131917+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
2026-04-08T16:05:49.276194+00:00 monitor updated for google-gemma-4-31b-it_v28
Inference service chaiml-bobo-creative-lr-68876-v1 ready after 253.46247673034668s
Pipeline stage VLLMDeployer completed in 260.80s
run pipeline stage %s
Running pipeline stage StressChecker
2026-04-08T16:05:53.699851+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
2026-04-08T16:05:59.620062+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
Received healthy response to inference request in 12.63709282875061s
Received healthy response to inference request in 4.882525682449341s
Received healthy response to inference request in 4.185794353485107s
Received healthy response to inference request in 4.281155824661255s
Received healthy response to inference request in 4.371833086013794s
Received healthy response to inference request in 4.2042131423950195s
Received healthy response to inference request in 4.204666614532471s
Received healthy response to inference request in 4.858266115188599s
Received healthy response to inference request in 4.154895782470703s
Received healthy response to inference request in 4.26915979385376s
2026-04-08T16:06:50.081572+00:00 monitor updated for google-gemma-4-31b-it_v28
Received healthy response to inference request in 4.471526145935059s
2026-04-08T16:06:54.175842+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
Received healthy response to inference request in 4.175513982772827s
2026-04-08T16:07:00.150301+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
Received healthy response to inference request in 4.217221260070801s
Received healthy response to inference request in 4.117532730102539s
Received healthy response to inference request in 4.521929502487183s
Received healthy response to inference request in 4.207247972488403s
Inference service chaiml-bobo-creative-lr2-5224-v1 ready after 334.358252286911s
Pipeline stage VLLMDeployer completed in 353.68s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.330422639846802s
Received healthy response to inference request in 4.4335198402404785s
Received healthy response to inference request in 11.503621339797974s
Received healthy response to inference request in 4.813175916671753s
Received healthy response to inference request in 11.63954472541809s
Received healthy response to inference request in 4.150347471237183s
Received healthy response to inference request in 4.197232246398926s
Received healthy response to inference request in 4.221250295639038s
Received healthy response to inference request in 4.400445222854614s
2026-04-08T16:07:50.581239+00:00 monitor updated for google-gemma-4-31b-it_v28
Received healthy response to inference request in 4.177430152893066s
2026-04-08T16:07:54.694589+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
Received healthy response to inference request in 4.152604341506958s
Received healthy response to inference request in 11.8175950050354s
2026-04-08T16:08:00.677361+00:00 monitor updated for chaiml-bobo-creative-lr_68876_v1
Received healthy response to inference request in 4.1863086223602295s
Received healthy response to inference request in 4.669449090957642s
Received healthy response to inference request in 11.591830968856812s
Received healthy response to inference request in 4.439723968505859s
Received healthy response to inference request in 4.1950483322143555s
Inference service google-gemma-4-31b-it-v28 ready after 335.13281655311584s
Pipeline stage VLLMDeployer completed in 337.70s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.1240599155426025s
Received healthy response to inference request in 11.994389295578003s
Received healthy response to inference request in 4.1125993728637695s
Received healthy response to inference request in 4.3610169887542725s
Received healthy response to inference request in 4.140762805938721s
Received healthy response to inference request in 11.720366954803467s
Received healthy response to inference request in 4.248918533325195s
Received healthy response to inference request in 4.150777101516724s
Received healthy response to inference request in 4.208745002746582s
30 requests
Received healthy response to inference request in 4.112037181854248s
0 failed requests
5th percentile: 4.153635489940643
10th percentile: 4.173452162742615
20th percentile: 4.186205768585205
30th percentile: 4.204530572891235
40th percentile: 4.213830757141113
50th percentile: 4.275157809257507
60th percentile: 4.365343427658081
Received healthy response to inference request in 11.668477773666382s
70th percentile: 4.435381078720093
80th percentile: 4.551433420181275
90th percentile: 5.558227586746225
Received healthy response to inference request in 4.813737630844116s
95th percentile: 11.834709239006042
99th percentile: 12.450708804130555
mean time: 5.106677516301473
Pipeline stage StressChecker completed in 175.04s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 4.1610023975372314s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.70s
Shutdown handler de-registered
chaiml-bobo-creative-lr_68876_v1 status is now deployed due to DeploymentManager action
2026-04-08T16:08:51.169696+00:00 monitor updated for google-gemma-4-31b-it_v28
Received healthy response to inference request in 4.299328804016113s
Received healthy response to inference request in 11.990406513214111s
2026-04-08T16:08:55.427486+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
Received healthy response to inference request in 4.171042442321777s
Received healthy response to inference request in 4.0722668170928955s
Received healthy response to inference request in 4.170721054077148s
Received healthy response to inference request in 4.127205848693848s
Received healthy response to inference request in 4.267432451248169s
Received healthy response to inference request in 4.132046222686768s
Received healthy response to inference request in 4.199783802032471s
Received healthy response to inference request in 4.746013879776001s
Received healthy response to inference request in 4.288292646408081s
Received healthy response to inference request in 4.220981597900391s
Received healthy response to inference request in 4.240430116653442s
Received healthy response to inference request in 4.123573064804077s
Received healthy response to inference request in 4.1411049365997314s
Received healthy response to inference request in 4.132135391235352s
Received healthy response to inference request in 4.14423942565918s
Received healthy response to inference request in 4.141566514968872s
Received healthy response to inference request in 4.917215347290039s
Received healthy response to inference request in 4.1535561084747314s
Received healthy response to inference request in 4.219464063644409s
Received healthy response to inference request in 4.373840808868408s
2026-04-08T16:09:51.599007+00:00 monitor updated for google-gemma-4-31b-it_v28
2026-04-08T16:09:55.847002+00:00 monitor updated for chaiml-bobo-creative-lr2_5224_v1
Received healthy response to inference request in 15.479552745819092s
Received healthy response to inference request in 15.606822729110718s
Received healthy response to inference request in 4.155206918716431s
Received healthy response to inference request in 4.8643670082092285s
Received healthy response to inference request in 4.234857082366943s
Received healthy response to inference request in 4.3405022621154785s
Received healthy response to inference request in 4.2436676025390625s
Received healthy response to inference request in 4.153642654418945s
Received healthy response to inference request in 4.461105585098267s
30 requests
Received healthy response to inference request in 4.15757417678833s
0 failed requests
5th percentile: 4.117756617069245
10th percentile: 4.139092516899109
20th percentile: 4.149125862121582
30th percentile: 4.159263753890992
40th percentile: 4.185445976257324
50th percentile: 4.220357179641724
60th percentile: 4.241725111007691
70th percentile: 4.2916034936904905
80th percentile: 4.813288259506225
90th percentile: 11.512442302703857
95th percentile: 11.716001188755035
99th percentile: 14.417585000991824
mean time: 5.384709692001342
Pipeline stage StressChecker completed in 179.61s
Received healthy response to inference request in 4.18746018409729s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.11s
Shutdown handler de-registered
chaiml-bobo-creative-lr2_5224_v1 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 4.28255820274353s
Received healthy response to inference request in 4.159440040588379s
Received healthy response to inference request in 4.502305507659912s
Received healthy response to inference request in 4.228703737258911s
Received healthy response to inference request in 4.192782640457153s
2026-04-08T16:10:52.060079+00:00 monitor updated for google-gemma-4-31b-it_v28
Received healthy response to inference request in 4.371943712234497s
Received healthy response to inference request in 4.168527126312256s
Received healthy response to inference request in 14.744744062423706s
Received healthy response to inference request in 4.225849866867065s
Received healthy response to inference request in 4.205397605895996s
Received healthy response to inference request in 4.302900075912476s
30 requests
0 failed requests
5th percentile: 4.1252078175544735
10th percentile: 4.131562185287476
20th percentile: 4.151158189773559
30th percentile: 4.158880281448364
40th percentile: 4.190653657913208
50th percentile: 4.223415732383728
60th percentile: 4.290694952011108
70th percentile: 4.372512841224671
80th percentile: 4.769684505462647
90th percentile: 11.747370910644532
95th percentile: 13.50529216527938
99th percentile: 15.356819915771485
mean time: 5.743265302975972
Pipeline stage StressChecker completed in 190.56s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.40s
Shutdown handler de-registered
google-gemma-4-31b-it_v28 status is now deployed due to DeploymentManager action
google-gemma-4-31b-it_v28 status is now inactive due to auto deactivation removed underperforming models