chaiml-pony-v1-q27b-lr1_56298

developer_uid: zonemercy
submission_id: chaiml-pony-v1-q27b-lr1_56298_v1
model_name: chaiml-pony-v1-q27b-lr1_56298_v1
model_group: ChaiML/pony-v1-q27b-lr1e
status: deployed
timestamp: 2026-03-18T16:23:45+00:00
num_battles: 2554
num_wins: 1360
celo_rating: 1316.57
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v1-q27b-lr1e4ep2r64g4
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v1-q27b-lr1_56298_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v1-q27b-lr1e4ep2r64g4
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-18
win_ratio: 0.5324980422866092
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '####', '<|im_end|>', '</s>', '<|user|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: Using quantization_mode: none
Starting job with name chaiml-pony-v1-q27b-lr1-56298-v1-uploader
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: Downloading snapshot of ChaiML/opusd-v4-q27b-lr1e4ep2r64g4...
Waiting for job on chaiml-pony-v1-q27b-lr1-56298-v1-uploader to finish
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: Using quantization_mode: none
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: Downloading snapshot of ChaiML/pony-v1-q27b-lr1e4ep2r64g4...
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: Downloaded in 41.934s
2026-03-18T16:09:36.313306+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: Downloaded in 40.464s
2026-03-18T16:09:47.117100+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: Processed model ChaiML/opusd-v4-q27b-lr1e4ep2r64g4 in 62.043s
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: creating bucket guanaco-vllm-models
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/generation_config.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/added_tokens.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/preprocessor_config.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/chat_template.jinja
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/README.md
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/model.safetensors.index.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/args.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/config.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/special_tokens_map.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/trainer_state.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/trainer_state.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/tokenizer_config.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/processor_config.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/training_args.bin
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/.gitattributes
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/vocab.json
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/merges.txt
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: Processed model ChaiML/pony-v1-q27b-lr1e4ep2r64g4 in 61.449s
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v1-q27b-lr1-56298-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v1-q27b-lr1-56298-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v1-q27b-lr1-56298-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v1-q27b-lr1-56298-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v1-q27b-lr1-56298-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v1-q27b-lr1-56298-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v1-q27b-lr1-56298-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v1-q27b-lr1-56298-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/special_tokens_map.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/.gitattributes
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/added_tokens.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/chat_template.jinja
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/args.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/preprocessor_config.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/generation_config.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/tokenizer_config.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/config.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/processor_config.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/training_args.bin
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/README.md
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/tokenizer.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/vocab.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/model.safetensors.index.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/trainer_state.json s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/trainer_state.json
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/merges.txt
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/model-00002-of-00002.safetensors
2026-03-18T16:10:36.458457+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/model-00002-of-00002.safetensors
2026-03-18T16:10:47.252134+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
2026-03-18T16:11:36.608131+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
2026-03-18T16:11:47.405086+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
2026-03-18T16:12:36.757390+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
2026-03-18T16:12:47.548500+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
2026-03-18T16:13:36.902749+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
2026-03-18T16:13:47.694246+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
2026-03-18T16:14:37.093290+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
2026-03-18T16:14:47.859227+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
chaiml-pony-v1-q27b-lr1-56298-v1-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v1-q27b-lr1-56298-v1/default/model-00001-of-00002.safetensors
Job chaiml-pony-v1-q27b-lr1-56298-v1-uploader completed after 408.67s with status: succeeded
Stopping job with name chaiml-pony-v1-q27b-lr1-56298-v1-uploader
Pipeline stage VLLMUploader completed in 409.44s
run pipeline stage %s
Running pipeline stage VLLMTemplater
2026-03-18T16:15:37.240149+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
Pipeline stage VLLMTemplater completed in 1.25s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v1-q27b-lr1-56298-v1
Waiting for inference service chaiml-pony-v1-q27b-lr1-56298-v1 to be ready
2026-03-18T16:15:48.008504+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
chaiml-opusd-v4-q27b-lr1-4495-v1-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-q27b-lr1-4495-v1/default/model-00001-of-00002.safetensors
Job chaiml-opusd-v4-q27b-lr1-4495-v1-uploader completed after 460.33s with status: succeeded
Stopping job with name chaiml-opusd-v4-q27b-lr1-4495-v1-uploader
Pipeline stage VLLMUploader completed in 461.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.53s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-opusd-v4-q27b-lr1-4495-v1
Waiting for inference service chaiml-opusd-v4-q27b-lr1-4495-v1 to be ready
2026-03-18T16:16:37.390687+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
2026-03-18T16:16:56.691543+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
2026-03-18T16:17:37.850094+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
2026-03-18T16:17:56.851828+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
Inference service chaiml-pony-v1-q27b-lr1-56298-v1 ready after 170.86943531036377s
Pipeline stage VLLMDeployer completed in 171.71s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-18T16:18:38.065023+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-18T16:18:57.069818+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
Inference service chaiml-opusd-v4-q27b-lr1-4495-v1 ready after 161.4627583026886s
Pipeline stage VLLMDeployer completed in 162.44s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-18T16:19:38.300542+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.82312798500061s
2026-03-18T16:19:57.246679+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 6.154016971588135s
Received healthy response to inference request in 10.940813064575195s
2026-03-18T16:20:38.553802+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-18T16:20:57.427072+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.549654483795166s
Received healthy response to inference request in 4.409103155136108s
Received healthy response to inference request in 2.2966408729553223s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.3926851749420166s
Received healthy response to inference request in 2.6829731464385986s
Received healthy response to inference request in 2.384229898452759s
Received healthy response to inference request in 2.3081820011138916s
Received healthy response to inference request in 4.47332763671875s
Received healthy response to inference request in 2.2695577144622803s
Received healthy response to inference request in 2.2791051864624023s
2026-03-18T16:21:38.726329+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.2011466026306152s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 7.271361351013184s
Received healthy response to inference request in 8.908389806747437s
Received healthy response to inference request in 4.475184679031372s
2026-03-18T16:21:57.605776+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
Received healthy response to inference request in 2.3864247798919678s
Received healthy response to inference request in 2.673412561416626s
Received healthy response to inference request in 2.426769495010376s
Received healthy response to inference request in 2.512688398361206s
Received healthy response to inference request in 2.6365487575531006s
Received healthy response to inference request in 2.276648759841919s
Received healthy response to inference request in 2.180128812789917s
Received healthy response to inference request in 2.3652281761169434s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.7820546627044678s
Received healthy response to inference request in 2.3406126499176025s
Received healthy response to inference request in 2.349766492843628s
Received healthy response to inference request in 2.239441394805908s
Received healthy response to inference request in 2.4193928241729736s
30 requests
8 failed requests
5th percentile: 2.2727486848831178
10th percentile: 2.278859543800354
20th percentile: 2.3414495944976808
30th percentile: 2.385766315460205
40th percentile: 2.423818826675415
50th percentile: 2.5931016206741333
60th percentile: 3.432874059677122
70th percentile: 6.980328822135918
80th percentile: 20.281586790084837
90th percentile: 20.325354480743407
95th percentile: 20.345952343940734
99th percentile: 20.373325431346892
mean time: 7.677257911364237
%s, retrying in %s seconds...
Received healthy response to inference request in 2.157155990600586s
Received healthy response to inference request in 2.1483590602874756s
Received healthy response to inference request in 2.245051383972168s
Received healthy response to inference request in 2.303481101989746s
2026-03-18T16:22:38.978542+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
Received healthy response to inference request in 2.32253360748291s
Received healthy response to inference request in 2.1329309940338135s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.2064578533172607s
Received healthy response to inference request in 2.395465850830078s
Received healthy response to inference request in 2.2412827014923096s
Received healthy response to inference request in 2.2691543102264404s
Received healthy response to inference request in 2.3351974487304688s
Received healthy response to inference request in 2.398242712020874s
Received healthy response to inference request in 2.242591619491577s
Received healthy response to inference request in 2.2069027423858643s
Received healthy response to inference request in 2.3153412342071533s
2026-03-18T16:22:57.879582+00:00 monitor updated for chaiml-pony-v1-q27b-lr1_56298_v1
Received healthy response to inference request in 2.2500829696655273s
Received healthy response to inference request in 2.3045101165771484s
Received healthy response to inference request in 2.3255932331085205s
Received healthy response to inference request in 2.359091281890869s
Received healthy response to inference request in 2.3415017127990723s
Received healthy response to inference request in 19.81424903869629s
Received healthy response to inference request in 2.569350481033325s
Received healthy response to inference request in 2.5562140941619873s
Received healthy response to inference request in 2.3706891536712646s
Received healthy response to inference request in 2.699599504470825s
Received healthy response to inference request in 2.5013105869293213s
Received healthy response to inference request in 2.7675371170043945s
Received healthy response to inference request in 2.2871346473693848s
Received healthy response to inference request in 2.316432237625122s
Received healthy response to inference request in 2.393738269805908s
Received healthy response to inference request in 2.3443443775177s
Received healthy response to inference request in 2.638967275619507s
Received healthy response to inference request in 2.364694833755493s
Received healthy response to inference request in 2.609088897705078s
Received healthy response to inference request in 2.2307522296905518s
Received healthy response to inference request in 2.283094644546509s
Received healthy response to inference request in 2.3347721099853516s
Received healthy response to inference request in 2.3165762424468994s
Received healthy response to inference request in 2.2130119800567627s
30 requests
9 failed requests
5th percentile: 2.2209950923919677
10th percentile: 2.2385724782943726
20th percentile: 2.3311041355133058
30th percentile: 2.358589696884155
40th percentile: 2.493025541305542
50th percentile: 2.73356831073761
Received healthy response to inference request in 2.2935633659362793s
60th percentile: 5.593655347824093
70th percentile: 19.95820240974426
80th percentile: 20.302517938613892
90th percentile: 20.32853443622589
95th percentile: 20.342439305782317
99th percentile: 20.36834329366684
mean time: 8.941604868570964
%s, retrying in %s seconds...
Received healthy response to inference request in 2.316895008087158s
Received healthy response to inference request in 2.224341630935669s
Received healthy response to inference request in 2.3226394653320312s
2026-03-18T16:23:39.207739+00:00 monitor updated for chaiml-opusd-v4-q27b-lr1_4495_v1
Received healthy response to inference request in 2.2094483375549316s
Received healthy response to inference request in 2.421182155609131s
30 requests
0 failed requests
5th percentile: 2.1523176789283753
10th percentile: 2.2015276670455934
Received healthy response to inference request in 2.1203718185424805s
20th percentile: 2.2423298358917236
30th percentile: 2.273191142082214
40th percentile: 2.2995140075683596
50th percentile: 2.3159587383270264
60th percentile: 2.3225759506225585
70th percentile: 2.3370887279510497
80th percentile: 2.3752989768981934
90th percentile: 2.5081145763397217
95th percentile: 2.5912066102027893
99th percentile: 2.6303025460243226
mean time: 2.3254098415374758
Pipeline stage StressChecker completed in 313.76s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
Received healthy response to inference request in 2.2534377574920654s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.10s
Shutdown handler de-registered
chaiml-pony-v1-q27b-lr1_56298_v1 status is now deployed due to DeploymentManager action