chaiml-bobo-creative-lr_65927

developer_uid: rirv938
submission_id: chaiml-bobo-creative-lr_65927_v1
model_name: chaiml-bobo-creative-lr_65927_v1
model_group: ChaiML/bobo_creative_lr1
status: inactive
timestamp: 2026-04-08T18:59:42+00:00
num_battles: 11087
num_wins: 6031
celo_rating: 1334.17
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/bobo_creative_lr1_full_run_fixed-step1403-merged
model_architecture: Gemma4ForConditionalGeneration
model_num_parameters: 28830461184.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-bobo-creative-lr_65927_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/bobo_creative_lr1_full_run_fixed-step1403-merged
model_size: 29B
ranking_group: single
us_pacific_date: 2026-04-08
win_ratio: 0.543970415802291
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 64, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_start|>', '###', 'You:', '<|redacted_im_end|>', '<turn|>', '<|turn>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|turn>system\nYou are "{bot_name}". Stay in character and respond only as "{bot_name}". Create an entertaining dialogue<turn|>\n', 'prompt_template': '', 'bot_template': '<|turn>model\n{message}<turn|>\n', 'user_template': '<|turn>user\n{message}<turn|>\n', 'response_template': '<|turn>model\n<|channel>thought\n<channel|>', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-bobo-creative-lr-65927-v1-uploader
chaiml-gembon-lr2-full-81509-v2-uploader: Using quantization_mode: none
Waiting for job on chaiml-bobo-creative-lr-65927-v1-uploader to finish
chaiml-gembon-lr2-full-81509-v2-uploader: Downloading snapshot of ChaiML/gembon_lr2_full_run_fixed-step1474-merged...
chaiml-gembon-lr1-full-94719-v1-uploader: Using quantization_mode: none
chaiml-gembon-lr1-full-94719-v1-uploader: Downloading snapshot of ChaiML/gembon_lr1_full_run_fixed-step1474-merged...
chaiml-bobo-creative-lr-65927-v1-uploader: Using quantization_mode: none
chaiml-bobo-creative-lr-65927-v1-uploader: Downloading snapshot of ChaiML/bobo_creative_lr1_full_run_fixed-step1403-merged...
chaiml-gembon-lr1-full-94719-v1-uploader: Downloaded in 24.517s
chaiml-gembon-lr2-full-81509-v2-uploader: Downloaded in 26.558s
2026-04-08T16:53:53.328652+00:00 monitor updated for chaiml-gembon-lr2-full-_81509_v2
chaiml-gembon-lr2-full-81509-v2-uploader: Processed model ChaiML/gembon_lr2_full_run_fixed-step1474-merged in 49.436s
chaiml-gembon-lr2-full-81509-v2-uploader: creating bucket guanaco-vllm-models
2026-04-08T16:53:58.609699+00:00 monitor updated for chaiml-gembon-lr1-full-_94719_v1
chaiml-gembon-lr2-full-81509-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr2-full-81509-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gembon-lr2-full-81509-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gembon-lr2-full-81509-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gembon-lr2-full-81509-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr2-full-81509-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gembon-lr2-full-81509-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr2-full-81509-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gembon-lr2-full-81509-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr2-full-81509-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gembon-lr2-full-81509-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr2-full-81509-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gembon-lr2-full-81509-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gembon-lr2-full-81509-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gembon-lr2-full-81509-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gembon-lr2-full-81509-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gembon-lr2-full-81509-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gembon-lr2-full-81509-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/.gitattributes
chaiml-gembon-lr1-full-94719-v1-uploader: Processed model ChaiML/gembon_lr1_full_run_fixed-step1474-merged in 47.844s
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model.safetensors.index.json
2026-04-08T16:54:03.899293+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
chaiml-gembon-lr1-full-94719-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/processor_config.json
chaiml-gembon-lr1-full-94719-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/tokenizer_config.json
chaiml-gembon-lr1-full-94719-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/generation_config.json
chaiml-gembon-lr1-full-94719-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/config.json
chaiml-gembon-lr1-full-94719-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/README.md
chaiml-gembon-lr1-full-94719-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/chat_template.jinja
chaiml-gembon-lr1-full-94719-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/args.json
chaiml-gembon-lr1-full-94719-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/tokenizer.json
chaiml-gembon-lr1-full-94719-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gembon-lr1-full-94719-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr1-full-94719-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gembon-lr1-full-94719-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr1-full-94719-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gembon-lr1-full-94719-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gembon-lr1-full-94719-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gembon-lr1-full-94719-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gembon-lr1-full-94719-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bobo-creative-lr-65927-v1-uploader: Processed model ChaiML/bobo_creative_lr1_full_run_fixed-step1403-merged in 49.443s
chaiml-gembon-lr1-full-94719-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bobo-creative-lr-65927-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gembon-lr1-full-94719-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default
chaiml-bobo-creative-lr-65927-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/config.json
chaiml-bobo-creative-lr-65927-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model.safetensors.index.json
chaiml-bobo-creative-lr-65927-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/README.md
chaiml-bobo-creative-lr-65927-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/chat_template.jinja
chaiml-bobo-creative-lr-65927-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/processor_config.json
chaiml-bobo-creative-lr-65927-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/args.json
chaiml-bobo-creative-lr-65927-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/tokenizer_config.json
chaiml-bobo-creative-lr-65927-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/.gitattributes
chaiml-bobo-creative-lr-65927-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/generation_config.json
chaiml-bobo-creative-lr-65927-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/tokenizer.json
chaiml-bobo-creative-lr-65927-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-bobo-creative-lr-65927-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-bobo-creative-lr-65927-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-bobo-creative-lr-65927-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-bobo-creative-lr-65927-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-bobo-creative-lr-65927-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-bobo-creative-lr-65927-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bobo-creative-lr-65927-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/chat_template.jinja
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/args.json
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/.gitattributes
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/model.safetensors.index.json
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/tokenizer_config.json
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/README.md
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00003-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00003-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/generation_config.json
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00007-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00007-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/config.json
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00004-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00004-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/processor_config.json
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00008-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00008-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/tokenizer.json
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00002-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00002-of-00013.safetensors
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00001-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00001-of-00013.safetensors
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00013-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00013-of-00013.safetensors
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00010-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00010-of-00013.safetensors
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00011-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00011-of-00013.safetensors
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00009-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00009-of-00013.safetensors
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00012-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00012-of-00013.safetensors
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00005-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00005-of-00013.safetensors
chaiml-gembon-lr2-full-81509-v2-uploader: cp /dev/shm/model_output/model-00006-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr2-full-81509-v2/default/model-00006-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00005-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00005-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00001-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00001-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00012-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00012-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00004-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00004-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00013-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00013-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00007-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00007-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00008-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00008-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00011-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00011-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00006-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00006-of-00013.safetensors
chaiml-gembon-lr1-full-94719-v1-uploader: cp /dev/shm/model_output/model-00003-of-00013.safetensors s3://guanaco-vllm-models/chaiml-gembon-lr1-full-94719-v1/default/model-00003-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/model-00009-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/model-00009-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/model-00011-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/model-00011-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/model-00001-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/model-00001-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/model-00008-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/model-00008-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/model-00010-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/model-00010-of-00013.safetensors
chaiml-bobo-creative-lr-65927-v1-uploader: cp /dev/shm/model_output/model-00003-of-00013.safetensors s3://guanaco-vllm-models/chaiml-bobo-creative-lr-65927-v1/default/model-00003-of-00013.safetensors
Job chaiml-gembon-lr2-full-81509-v2-uploader completed after 97.75s with status: succeeded
Stopping job with name chaiml-gembon-lr2-full-81509-v2-uploader
Pipeline stage VLLMUploader completed in 99.76s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.69s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.61s
run pipeline stage %s
Job chaiml-gembon-lr1-full-94719-v1-uploader completed after 97.45s with status: succeeded
Running pipeline stage VLLMDeployer
Stopping job with name chaiml-gembon-lr1-full-94719-v1-uploader
Creating inference service chaiml-gembon-lr2-full-81509-v2
Pipeline stage VLLMUploader completed in 99.33s
run pipeline stage %s
Waiting for inference service chaiml-gembon-lr2-full-81509-v2 to be ready
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.78s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Job chaiml-bobo-creative-lr-65927-v1-uploader completed after 95.94s with status: succeeded
Pipeline stage VLLMTemplater completed in 1.46s
Stopping job with name chaiml-bobo-creative-lr-65927-v1-uploader
run pipeline stage %s
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploader completed in 98.22s
run pipeline stage %s
Creating inference service chaiml-gembon-lr1-full-94719-v1
Running pipeline stage VLLMUploaderAMD
Waiting for inference service chaiml-gembon-lr1-full-94719-v1 to be ready
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.99s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.20s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-bobo-creative-lr-65927-v1
Waiting for inference service chaiml-bobo-creative-lr-65927-v1 to be ready
2026-04-08T16:54:53.693549+00:00 monitor updated for chaiml-gembon-lr2-full-_81509_v2
2026-04-08T16:54:59.026603+00:00 monitor updated for chaiml-gembon-lr1-full-_94719_v1
2026-04-08T16:55:04.442031+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
2026-04-08T16:55:54.183211+00:00 monitor updated for chaiml-gembon-lr2-full-_81509_v2
2026-04-08T16:55:59.550792+00:00 monitor updated for chaiml-gembon-lr1-full-_94719_v1
2026-04-08T16:56:04.992476+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
2026-04-08T16:56:55.184609+00:00 monitor updated for chaiml-gembon-lr2-full-_81509_v2
2026-04-08T16:57:00.212555+00:00 monitor updated for chaiml-gembon-lr1-full-_94719_v1
2026-04-08T16:57:05.614421+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
2026-04-08T16:57:55.743139+00:00 monitor updated for chaiml-gembon-lr2-full-_81509_v2
2026-04-08T16:58:00.788652+00:00 monitor updated for chaiml-gembon-lr1-full-_94719_v1
2026-04-08T16:58:06.416602+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
Inference service chaiml-gembon-lr1-full-94719-v1 ready after 232.63477778434753s
Pipeline stage VLLMDeployer completed in 235.04s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service chaiml-gembon-lr2-full-81509-v2 ready after 242.70060420036316s
Pipeline stage VLLMDeployer completed in 245.09s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 12.620926141738892s
Received healthy response to inference request in 11.986237287521362s
2026-04-08T16:58:56.595494+00:00 monitor updated for chaiml-gembon-lr2-full-_81509_v2
Received healthy response to inference request in 4.36197829246521s
2026-04-08T16:59:01.401474+00:00 monitor updated for chaiml-gembon-lr1-full-_94719_v1
Received healthy response to inference request in 4.367152690887451s
Received healthy response to inference request in 4.211767673492432s
2026-04-08T16:59:07.054100+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
Received healthy response to inference request in 12.144373178482056s
Received healthy response to inference request in 11.911978483200073s
Received healthy response to inference request in 13.167856216430664s
Received healthy response to inference request in 4.354984760284424s
Received healthy response to inference request in 4.816038131713867s
Received healthy response to inference request in 13.188946008682251s
Received healthy response to inference request in 4.2379069328308105s
Received healthy response to inference request in 4.2002034187316895s
Received healthy response to inference request in 4.2879722118377686s
Received healthy response to inference request in 4.183773994445801s
Received healthy response to inference request in 4.253221750259399s
Received healthy response to inference request in 4.885531187057495s
Received healthy response to inference request in 4.228927850723267s
Received healthy response to inference request in 4.088598251342773s
Received healthy response to inference request in 4.24318265914917s
Received healthy response to inference request in 4.160508632659912s
2026-04-08T16:59:57.488035+00:00 monitor updated for chaiml-gembon-lr2-full-_81509_v2
Received healthy response to inference request in 4.1966986656188965s
Received healthy response to inference request in 4.188430547714233s
2026-04-08T17:00:02.233582+00:00 monitor updated for chaiml-gembon-lr1-full-_94719_v1
Received healthy response to inference request in 4.7689549922943115s
Received healthy response to inference request in 4.413576364517212s
2026-04-08T17:00:07.678638+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
Received healthy response to inference request in 4.350956201553345s
Received healthy response to inference request in 4.236730575561523s
Received healthy response to inference request in 4.24978232383728s
Received healthy response to inference request in 4.129538297653198s
Received healthy response to inference request in 4.823972940444946s
Received healthy response to inference request in 4.914859056472778s
Received healthy response to inference request in 4.095962285995483s
Received healthy response to inference request in 4.192986726760864s
Received healthy response to inference request in 13.077704429626465s
Received healthy response to inference request in 4.306109189987183s
Received healthy response to inference request in 4.395174741744995s
Received healthy response to inference request in 4.223969221115112s
Received healthy response to inference request in 4.309651613235474s
Received healthy response to inference request in 4.3142499923706055s
Received healthy response to inference request in 4.171973466873169s
Received healthy response to inference request in 4.117637634277344s
Received healthy response to inference request in 4.246863842010498s
Received healthy response to inference request in 4.361995697021484s
2026-04-08T17:00:58.023477+00:00 monitor updated for chaiml-gembon-lr2-full-_81509_v2
Received healthy response to inference request in 4.433671236038208s
Received healthy response to inference request in 4.243293523788452s
2026-04-08T17:01:02.773178+00:00 monitor updated for chaiml-gembon-lr1-full-_94719_v1
Received healthy response to inference request in 4.183326482772827s
Received healthy response to inference request in 4.2136359214782715s
2026-04-08T17:01:08.196029+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
Received healthy response to inference request in 4.116386413574219s
Received healthy response to inference request in 4.174461126327515s
Received healthy response to inference request in 4.185428619384766s
Received healthy response to inference request in 4.272953033447266s
Received healthy response to inference request in 4.222185850143433s
Received healthy response to inference request in 4.242143630981445s
Received healthy response to inference request in 4.271848440170288s
Received healthy response to inference request in 4.365145206451416s
Received healthy response to inference request in 4.297393083572388s
Received healthy response to inference request in 4.745048999786377s
30 requests
0 failed requests
5th percentile: 4.105716192722321
10th percentile: 4.128348231315613
20th percentile: 4.181911420822144
30th percentile: 4.198038411140442
40th percentile: 4.219835901260376
Received healthy response to inference request in 4.1698830127716064s
50th percentile: 4.242718577384949
60th percentile: 4.309365510940552
70th percentile: 4.362940549850464
80th percentile: 4.4798708915710455
90th percentile: 5.614570999145518
95th percentile: 12.301899695396422
99th percentile: 13.024220247268678
mean time: 5.131136393547058
Pipeline stage StressChecker completed in 177.52s
run pipeline stage %s
Received healthy response to inference request in 4.2142839431762695s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.82s
Shutdown handler de-registered
chaiml-gembon-lr1-full-_94719_v1 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 4.296135425567627s
30 requests
0 failed requests
5th percentile: 4.17082371711731
10th percentile: 4.182191181182861
20th percentile: 4.210766887664795
30th percentile: 4.235213208198547
40th percentile: 4.248614931106568
50th percentile: 4.279910326004028
60th percentile: 4.302296495437622
70th percentile: 4.367041754722595
80th percentile: 4.7783716201782225
90th percentile: 12.002050876617432
95th percentile: 12.657705366611477
99th percentile: 13.141712198257446
mean time: 5.423432556788127
Pipeline stage StressChecker completed in 183.18s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.16s
Shutdown handler de-registered
chaiml-gembon-lr2-full-_81509_v2 status is now deployed due to DeploymentManager action
2026-04-08T17:02:08.733090+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
Inference service chaiml-bobo-creative-lr-65927-v1 ready after 496.3755033016205s
Pipeline stage VLLMDeployer completed in 498.50s
run pipeline stage %s
Running pipeline stage StressChecker
2026-04-08T17:03:09.048024+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
Received healthy response to inference request in 11.88446855545044s
Received healthy response to inference request in 4.350507020950317s
Received healthy response to inference request in 4.145624399185181s
Received healthy response to inference request in 12.94531512260437s
Received healthy response to inference request in 4.325804948806763s
Received healthy response to inference request in 4.344546318054199s
Received healthy response to inference request in 4.64276123046875s
Received healthy response to inference request in 4.104601621627808s
Received healthy response to inference request in 4.184538841247559s
Received healthy response to inference request in 4.084338665008545s
2026-04-08T17:04:09.384933+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
Received healthy response to inference request in 5.454366445541382s
Received healthy response to inference request in 4.181675434112549s
Received healthy response to inference request in 4.08908486366272s
Received healthy response to inference request in 4.180004119873047s
Received healthy response to inference request in 4.3992767333984375s
Received healthy response to inference request in 4.346427917480469s
Received healthy response to inference request in 4.211591005325317s
Received healthy response to inference request in 4.190739393234253s
Received healthy response to inference request in 4.052616596221924s
Received healthy response to inference request in 4.116227865219116s
Received healthy response to inference request in 4.385716915130615s
Received healthy response to inference request in 4.265567302703857s
Received healthy response to inference request in 4.179703712463379s
2026-04-08T17:05:09.671070+00:00 monitor updated for chaiml-bobo-creative-lr_65927_v1
Received healthy response to inference request in 4.196671962738037s
Received healthy response to inference request in 4.279670715332031s
Received healthy response to inference request in 4.671552658081055s
Received healthy response to inference request in 4.184163570404053s
Received healthy response to inference request in 4.2363903522491455s
Received healthy response to inference request in 4.342523813247681s
Received healthy response to inference request in 4.58139967918396s
30 requests
0 failed requests
5th percentile: 4.086474454402923
10th percentile: 4.103049945831299
20th percentile: 4.172887849807739
30th percentile: 4.183417129516601
40th percentile: 4.194298934936524
50th percentile: 4.2509788274765015
60th percentile: 4.33249249458313
70th percentile: 4.347651648521423
80th percentile: 4.435701322555542
90th percentile: 4.749834036827089
95th percentile: 8.990922605991345
99th percentile: 12.637669618129731
mean time: 4.851929259300232
Pipeline stage StressChecker completed in 154.87s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.48s
Shutdown handler de-registered
chaiml-bobo-creative-lr_65927_v1 status is now deployed due to DeploymentManager action
chaiml-bobo-creative-lr_65927_v1 status is now inactive due to auto deactivation removed underperforming models