developer_uid: chai_backend_admin
submission_id: chaiml-pony-d3b-mv1-wi_84391_v12
model_name: chaiml-pony-d3b-mv1-wi_84391_v12
model_group: ChaiML/pony-d3b-mv1-wina
status: torndown
timestamp: 2026-04-01T00:01:56+00:00
num_battles: 10667
num_wins: 5626
celo_rating: 1314.84
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8
model_architecture: Qwen3_5MoeForConditionalGeneration
model_num_parameters: 33753909248.0
best_of: 16
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3b-mv1-wi_84391_v12
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8
model_size: 34B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5274210180931845
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.8, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|im_end|>', '<|user|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 16, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3b-mv1-wi-84391-v12-uploader
Waiting for job on chaiml-pony-d3b-mv1-wi-84391-v12-uploader to finish
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: Using quantization_mode: fp8
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: Checking if ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: Downloading snapshot of ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8-FP8...
2026-03-28T20:53:42.426359+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: Downloaded in 25.163s
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: Processed model ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8 in 27.950s
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-wi-84391-v12-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3b-mv1-wi-84391-v12-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-wi-84391-v12-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-wi-84391-v12-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-wi-84391-v12-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-wi-84391-v12-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3b-mv1-wi-84391-v12-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3b-mv1-wi-84391-v12-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default/.gitattributes
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default/recipe.yaml
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default/tokenizer_config.json
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default/config.json
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default/chat_template.jinja
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default/generation_config.json
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default/tokenizer.json
2026-03-28T20:54:42.506579+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
chaiml-pony-d3b-mv1-wi-84391-v12-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-wi-84391-v12/default/model.safetensors
Job chaiml-pony-d3b-mv1-wi-84391-v12-uploader completed after 133.11s with status: succeeded
Stopping job with name chaiml-pony-d3b-mv1-wi-84391-v12-uploader
Pipeline stage VLLMUploader completed in 133.54s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.08s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.40s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3b-mv1-wi-84391-v12
Waiting for inference service chaiml-pony-d3b-mv1-wi-84391-v12 to be ready
2026-03-28T20:55:42.588606+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
2026-03-28T20:56:42.695976+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
2026-03-28T20:57:42.826406+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
Inference service chaiml-pony-d3b-mv1-wi-84391-v12 ready after 170.55654978752136s
Pipeline stage VLLMDeployer completed in 171.03s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T20:58:42.926778+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 11.932019233703613s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.970539569854736s
2026-03-28T20:59:43.009752+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 6.769547939300537s
Received healthy response to inference request in 5.985517263412476s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T21:00:43.122732+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
Received healthy response to inference request in 12.339008569717407s
Received healthy response to inference request in 2.2385799884796143s
Received healthy response to inference request in 2.8734543323516846s
Received healthy response to inference request in 2.227020740509033s
Received healthy response to inference request in 4.421529531478882s
Received healthy response to inference request in 2.681626796722412s
Received healthy response to inference request in 2.3353469371795654s
Received healthy response to inference request in 5.590733289718628s
Received healthy response to inference request in 2.489044427871704s
Received healthy response to inference request in 3.1152567863464355s
Received healthy response to inference request in 2.265873432159424s
Received healthy response to inference request in 2.2193710803985596s
Received healthy response to inference request in 2.2088892459869385s
Received healthy response to inference request in 3.1402018070220947s
Received healthy response to inference request in 2.3357255458831787s
Received healthy response to inference request in 2.262983560562134s
Received healthy response to inference request in 2.3553974628448486s
Received healthy response to inference request in 2.264721393585205s
2026-03-28T21:01:43.206244+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
Received healthy response to inference request in 3.0603339672088623s
30 requests
7 failed requests
5th percentile: 2.2228134274482727
10th percentile: 2.237424063682556
20th percentile: 2.26564302444458
30th percentile: 2.3494958877563477
40th percentile: 2.7967233180999758
50th percentile: 3.127729296684265
60th percentile: 5.218617057800292
70th percentile: 8.318289327621445
80th percentile: 20.113153076171876
90th percentile: 20.117854356765747
95th percentile: 20.132919335365294
99th percentile: 20.135741810798645
mean time: 7.7648359934488935
%s, retrying in %s seconds...
Received healthy response to inference request in 2.292804002761841s
Received healthy response to inference request in 2.1438376903533936s
Received healthy response to inference request in 2.2690656185150146s
Received healthy response to inference request in 2.1651158332824707s
Received healthy response to inference request in 2.164774179458618s
Received healthy response to inference request in 2.146477222442627s
Received healthy response to inference request in 2.1038947105407715s
Received healthy response to inference request in 2.3411238193511963s
Received healthy response to inference request in 2.2683966159820557s
Received healthy response to inference request in 2.179694890975952s
Received healthy response to inference request in 2.1957242488861084s
Received healthy response to inference request in 2.1608991622924805s
Received healthy response to inference request in 2.232003688812256s
Received healthy response to inference request in 2.307063341140747s
Received healthy response to inference request in 2.556117296218872s
Received healthy response to inference request in 2.2252750396728516s
Received healthy response to inference request in 2.6965739727020264s
Received healthy response to inference request in 2.2804372310638428s
Received healthy response to inference request in 2.3096601963043213s
Received healthy response to inference request in 2.224470853805542s
Received healthy response to inference request in 2.255305290222168s
Received healthy response to inference request in 2.568901300430298s
Received healthy response to inference request in 2.480125904083252s
Received healthy response to inference request in 2.5060789585113525s
2026-03-28T21:02:43.298171+00:00 monitor updated for chaiml-pony-d3b-mv1-wi_84391_v12
Received healthy response to inference request in 2.406766414642334s
Received healthy response to inference request in 2.2404074668884277s
Received healthy response to inference request in 2.4221012592315674s
Received healthy response to inference request in 2.239577054977417s
Received healthy response to inference request in 2.3995206356048584s
Received healthy response to inference request in 2.2818591594696045s
30 requests
0 failed requests
5th percentile: 2.1450254797935484
10th percentile: 2.159456968307495
20th percentile: 2.1767790794372557
30th percentile: 2.2250337839126586
40th percentile: 2.2400753021240236
50th percentile: 2.268731117248535
60th percentile: 2.286237096786499
70th percentile: 2.319099283218384
80th percentile: 2.409833383560181
90th percentile: 2.5110827922821044
95th percentile: 2.5631484985351562
99th percentile: 2.659548897743225
mean time: 2.3021351019541423
Pipeline stage StressChecker completed in 306.77s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.32s
Shutdown handler de-registered
chaiml-pony-d3b-mv1-wi_84391_v12 status is now deployed due to DeploymentManager action
chaiml-pony-d3b-mv1-wi_84391_v12 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3b-mv1-wi_84391_v12 status is now torndown due to DeploymentManager action