developer_uid: zonemercy
submission_id: chaiml-pony-v3-q27b-lr5_22882_v4
model_name: chaiml-pony-v3-q27b-lr5_22882_v4
model_group: ChaiML/pony-v3-q27b-lr5e
status: torndown
timestamp: 2026-03-31T12:51:20+00:00
num_battles: 11952
num_wins: 6268
celo_rating: 1311.07
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v3-q27b-lr5e6ep1g8
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v3-q27b-lr5_22882_v4
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v3-q27b-lr5e6ep1g8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5244310575635877
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.8, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|user|>', '</s>', '<|assistant|>', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v3-q27b-lr5-22882-v4-uploader
Waiting for job on chaiml-pony-v3-q27b-lr5-22882-v4-uploader to finish
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: Using quantization_mode: fp8
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: Checking if ChaiML/pony-v3-q27b-lr5e6ep1g8-FP8 already exists in ChaiML
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: Downloading snapshot of ChaiML/pony-v3-q27b-lr5e6ep1g8-FP8...
2026-03-28T10:49:15.729012+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: Downloaded in 23.331s
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: Processed model ChaiML/pony-v3-q27b-lr5e6ep1g8 in 25.797s
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v3-q27b-lr5-22882-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v3-q27b-lr5-22882-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v3-q27b-lr5-22882-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default/tokenizer_config.json
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default/config.json
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default/chat_template.jinja
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default/generation_config.json
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default/recipe.yaml
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default/.gitattributes
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default/tokenizer.json
2026-03-28T10:50:15.817791+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
chaiml-pony-v3-q27b-lr5-22882-v4-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v4/default/model.safetensors
Job chaiml-pony-v3-q27b-lr5-22882-v4-uploader completed after 132.72s with status: succeeded
Stopping job with name chaiml-pony-v3-q27b-lr5-22882-v4-uploader
Pipeline stage VLLMUploader completed in 133.20s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.16s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.04s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v3-q27b-lr5-22882-v4
Waiting for inference service chaiml-pony-v3-q27b-lr5-22882-v4 to be ready
2026-03-28T10:51:15.907063+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
2026-03-28T10:52:16.001898+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
2026-03-28T10:53:16.102368+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
Inference service chaiml-pony-v3-q27b-lr5-22882-v4 ready after 181.28125667572021s
Pipeline stage VLLMDeployer completed in 181.75s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T10:54:16.201747+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T10:55:16.299949+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 11.900973558425903s
Received healthy response to inference request in 4.377349615097046s
Received healthy response to inference request in 14.991558313369751s
Received healthy response to inference request in 1.9589762687683105s
Received healthy response to inference request in 2.5828018188476562s
2026-03-28T10:56:16.403955+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.0020740032196045s
Received healthy response to inference request in 2.1185708045959473s
Received healthy response to inference request in 1.862433671951294s
Received healthy response to inference request in 1.885087013244629s
Received healthy response to inference request in 1.9486510753631592s
Received healthy response to inference request in 4.532553195953369s
Received healthy response to inference request in 2.5387659072875977s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.049774408340454s
Received healthy response to inference request in 2.0560646057128906s
Received healthy response to inference request in 2.5565381050109863s
2026-03-28T10:57:16.507335+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
Received healthy response to inference request in 11.091022253036499s
Received healthy response to inference request in 1.94285249710083s
Received healthy response to inference request in 1.9134747982025146s
Received healthy response to inference request in 1.899928331375122s
Received healthy response to inference request in 1.9829888343811035s
Received healthy response to inference request in 1.9991278648376465s
Received healthy response to inference request in 2.14615535736084s
30 requests
8 failed requests
5th percentile: 1.8917656064033508
10th percentile: 1.9121201515197754
20th percentile: 1.9569112300872802
30th percentile: 2.001190161705017
40th percentile: 2.0935683250427246
50th percentile: 2.547652006149292
60th percentile: 4.439431047439575
70th percentile: 12.828148984909049
80th percentile: 20.123726558685302
90th percentile: 20.129254174232482
95th percentile: 20.13134559392929
99th percentile: 20.138117246627807
mean time: 8.111905535062155
%s, retrying in %s seconds...
Received healthy response to inference request in 1.8264679908752441s
Received healthy response to inference request in 1.7828574180603027s
Received healthy response to inference request in 2.0182993412017822s
Received healthy response to inference request in 2.023366928100586s
Received healthy response to inference request in 1.8993816375732422s
Received healthy response to inference request in 2.1765780448913574s
Received healthy response to inference request in 1.777550458908081s
Received healthy response to inference request in 2.0526907444000244s
Received healthy response to inference request in 1.8341560363769531s
Received healthy response to inference request in 1.8591821193695068s
Received healthy response to inference request in 1.8558025360107422s
Received healthy response to inference request in 1.8519656658172607s
Received healthy response to inference request in 2.6135482788085938s
Received healthy response to inference request in 2.1083414554595947s
Received healthy response to inference request in 2.078902006149292s
Received healthy response to inference request in 1.855968952178955s
Received healthy response to inference request in 2.0155506134033203s
Received healthy response to inference request in 1.975538969039917s
2026-03-28T10:58:16.660556+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v4
Received healthy response to inference request in 2.131307363510132s
Received healthy response to inference request in 1.9094204902648926s
Received healthy response to inference request in 1.8820738792419434s
Received healthy response to inference request in 1.9480791091918945s
Received healthy response to inference request in 1.952131748199463s
Received healthy response to inference request in 2.1046574115753174s
Received healthy response to inference request in 2.0956451892852783s
Received healthy response to inference request in 2.061940908432007s
Received healthy response to inference request in 1.967634677886963s
Received healthy response to inference request in 2.017076253890991s
Received healthy response to inference request in 2.0022735595703125s
Received healthy response to inference request in 1.9733173847198486s
30 requests
0 failed requests
5th percentile: 1.8024821758270264
10th percentile: 1.8333872318267823
20th percentile: 1.8559356689453126
30th percentile: 1.8941893100738525
40th percentile: 1.9505106925964355
50th percentile: 1.9744281768798828
60th percentile: 2.0161608695983886
70th percentile: 2.0321640729904176
80th percentile: 2.0822506427764895
90th percentile: 2.1106380462646483
95th percentile: 2.1562062382698057
99th percentile: 2.486826910972596
mean time: 1.9883902390797934
Pipeline stage StressChecker completed in 310.30s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.72s
Shutdown handler de-registered
chaiml-pony-v3-q27b-lr5_22882_v4 status is now deployed due to DeploymentManager action
chaiml-pony-v3-q27b-lr5_22882_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v3-q27b-lr5_22882_v4 status is now torndown due to DeploymentManager action