developer_uid: zonemercy
submission_id: chaiml-pony-d3b-mv1-win_84391_v4
model_name: chaiml-pony-d3b-mv1-win_84391_v4
model_group: ChaiML/pony-d3b-mv1-wina
status: torndown
timestamp: 2026-03-31T03:51:06+00:00
num_battles: 10613
num_wins: 5442
celo_rating: 1301.98
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8
model_architecture: Qwen3_5MoeForConditionalGeneration
model_num_parameters: 33753909248.0
best_of: 16
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3b-mv1-win_84391_v4
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8
model_size: 34B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5127673607839442
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.8, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_end|>', '<|user|>', '####', '<|assistant|>'], 'max_input_tokens': 2048, 'best_of': 16, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3b-mv1-win-84391-v4-uploader
Waiting for job on chaiml-pony-d3b-mv1-win-84391-v4-uploader to finish
chaiml-pony-d3b-mv1-win-84391-v4-uploader: Using quantization_mode: fp8
chaiml-pony-d3b-mv1-win-84391-v4-uploader: Checking if ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-pony-d3b-mv1-win-84391-v4-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-d3b-mv1-win-84391-v4-uploader: Downloading snapshot of ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8-FP8...
2026-03-28T01:23:52.995628+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
chaiml-pony-d3b-mv1-win-84391-v4-uploader: Downloaded in 34.563s
chaiml-pony-d3b-mv1-win-84391-v4-uploader: Processed model ChaiML/pony-d3b-mv1-winall-q35b-lr5e6ep2g8 in 37.074s
chaiml-pony-d3b-mv1-win-84391-v4-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3b-mv1-win-84391-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-win-84391-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3b-mv1-win-84391-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3b-mv1-win-84391-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3b-mv1-win-84391-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-win-84391-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3b-mv1-win-84391-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-win-84391-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3b-mv1-win-84391-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-win-84391-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3b-mv1-win-84391-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-win-84391-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3b-mv1-win-84391-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3b-mv1-win-84391-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3b-mv1-win-84391-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3b-mv1-win-84391-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3b-mv1-win-84391-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3b-mv1-win-84391-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default
chaiml-pony-d3b-mv1-win-84391-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default/config.json
chaiml-pony-d3b-mv1-win-84391-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default/chat_template.jinja
chaiml-pony-d3b-mv1-win-84391-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default/tokenizer_config.json
chaiml-pony-d3b-mv1-win-84391-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default/generation_config.json
chaiml-pony-d3b-mv1-win-84391-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default/.gitattributes
chaiml-pony-d3b-mv1-win-84391-v4-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default/recipe.yaml
chaiml-pony-d3b-mv1-win-84391-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default/tokenizer.json
chaiml-pony-d3b-mv1-win-84391-v4-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-win-84391-v4/default/model.safetensors
2026-03-28T01:24:53.078105+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
Job chaiml-pony-d3b-mv1-win-84391-v4-uploader completed after 124.63s with status: succeeded
Stopping job with name chaiml-pony-d3b-mv1-win-84391-v4-uploader
Pipeline stage VLLMUploader completed in 125.19s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.46s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3b-mv1-win-84391-v4
Waiting for inference service chaiml-pony-d3b-mv1-win-84391-v4 to be ready
2026-03-28T01:25:53.167514+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
2026-03-28T01:26:53.259549+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
2026-03-28T01:27:54.179326+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
Inference service chaiml-pony-d3b-mv1-win-84391-v4 ready after 191.05220246315002s
Pipeline stage VLLMDeployer completed in 191.59s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T01:28:54.496649+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 13.998718738555908s
Received healthy response to inference request in 2.060079336166382s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.736313819885254s
2026-03-28T01:29:54.610372+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 5.730299949645996s
Received healthy response to inference request in 2.577662467956543s
2026-03-28T01:30:54.706691+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
Received healthy response to inference request in 2.6570229530334473s
Failed to get request counts for guanaco-submitter. Falling back to default
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.7342536449432373s
Received healthy response to inference request in 5.7670440673828125s
Received healthy response to inference request in 2.2042012214660645s
Received healthy response to inference request in 5.746693134307861s
2026-03-28T01:31:54.811911+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
Received healthy response to inference request in 2.668006658554077s
Received healthy response to inference request in 2.158552885055542s
Received healthy response to inference request in 2.1178481578826904s
Received healthy response to inference request in 2.219005584716797s
Received healthy response to inference request in 2.133530616760254s
Received healthy response to inference request in 2.4054737091064453s
Received healthy response to inference request in 2.1429219245910645s
Received healthy response to inference request in 2.2804949283599854s
Received healthy response to inference request in 2.168881893157959s
Received healthy response to inference request in 2.421618938446045s
Received healthy response to inference request in 2.2158663272857666s
Received healthy response to inference request in 2.450802803039551s
30 requests
8 failed requests
5th percentile: 2.124905264377594
10th percentile: 2.1419827938079834
20th percentile: 2.1971373558044434
30th percentile: 2.262048125267029
40th percentile: 2.4391292572021483
50th percentile: 2.662514805793762
60th percentile: 3.9339082717895466
70th percentile: 8.236546468734717
80th percentile: 20.128302001953124
90th percentile: 20.151758980751037
95th percentile: 20.172723269462587
99th percentile: 20.688436169624328
mean time: 7.849945155779521
%s, retrying in %s seconds...
Received healthy response to inference request in 2.6573853492736816s
Received healthy response to inference request in 2.2034366130828857s
Received healthy response to inference request in 2.1029117107391357s
Received healthy response to inference request in 2.1283833980560303s
Received healthy response to inference request in 2.2040727138519287s
Received healthy response to inference request in 2.2455568313598633s
Received healthy response to inference request in 2.1712727546691895s
Received healthy response to inference request in 2.113424301147461s
Received healthy response to inference request in 2.1929404735565186s
Received healthy response to inference request in 2.087299108505249s
upstream connect error or disconnect/reset before headers. reset reason: connection termination
Received unhealthy response to inference request!
Received healthy response to inference request in 2.1791515350341797s
Received healthy response to inference request in 2.1100428104400635s
Received healthy response to inference request in 2.434516668319702s
2026-03-28T01:32:54.987489+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
Received healthy response to inference request in 2.199772357940674s
Received healthy response to inference request in 2.137395143508911s
Received healthy response to inference request in 2.191544532775879s
Received healthy response to inference request in 2.271629571914673s
Received healthy response to inference request in 2.2650034427642822s
Received healthy response to inference request in 2.1183547973632812s
Received healthy response to inference request in 2.121720314025879s
Received healthy response to inference request in 2.58424711227417s
Received healthy response to inference request in 2.229433536529541s
Received healthy response to inference request in 2.201475143432617s
Received healthy response to inference request in 2.199371814727783s
Received healthy response to inference request in 2.1590054035186768s
Received healthy response to inference request in 2.2070534229278564s
Received healthy response to inference request in 2.612428903579712s
Received healthy response to inference request in 2.2000319957733154s
Received healthy response to inference request in 2.6301259994506836s
30 requests
1 failed requests
5th percentile: 2.0943247795104982
10th percentile: 2.1093297004699707
20th percentile: 2.1210472106933596
30th percentile: 2.152522325515747
40th percentile: 2.186587333679199
50th percentile: 2.1995720863342285
60th percentile: 2.202259731292725
70th percentile: 2.2137674570083616
80th percentile: 2.2663286685943604
90th percentile: 2.5870652914047243
95th percentile: 2.6221623063087462
99th percentile: 2.6494801378250123
mean time: 2.1756588856379193
%s, retrying in %s seconds...
Received healthy response to inference request in 2.4500370025634766s
Received healthy response to inference request in 2.3502309322357178s
Received healthy response to inference request in 2.0769946575164795s
Received healthy response to inference request in 2.3749001026153564s
Received healthy response to inference request in 2.601588487625122s
Received healthy response to inference request in 2.1861186027526855s
Received healthy response to inference request in 2.4525816440582275s
Received healthy response to inference request in 2.0839574337005615s
Received healthy response to inference request in 2.127275228500366s
2026-03-28T01:33:55.098631+00:00 monitor updated for chaiml-pony-d3b-mv1-win_84391_v4
Received healthy response to inference request in 2.1310977935791016s
Received healthy response to inference request in 2.094195604324341s
Received healthy response to inference request in 2.6295578479766846s
Received healthy response to inference request in 2.1609878540039062s
Received healthy response to inference request in 2.116550922393799s
Received healthy response to inference request in 2.428941011428833s
Received healthy response to inference request in 2.1813957691192627s
Received healthy response to inference request in 2.1690680980682373s
Received healthy response to inference request in 2.3350307941436768s
Received healthy response to inference request in 2.2276337146759033s
Received healthy response to inference request in 2.4176671504974365s
Received healthy response to inference request in 2.110224962234497s
Received healthy response to inference request in 2.4538931846618652s
Received healthy response to inference request in 2.6774797439575195s
Received healthy response to inference request in 2.32487154006958s
Received healthy response to inference request in 2.2196362018585205s
Retrying (%r) after connection broken by '%r': %s
Received healthy response to inference request in 2.303426504135132s
Received healthy response to inference request in 2.241619348526001s
Received healthy response to inference request in 2.218644142150879s
Received healthy response to inference request in 2.7471563816070557s
Received healthy response to inference request in 2.1209802627563477s
30 requests
0 failed requests
5th percentile: 2.0885646104812623
10th percentile: 2.1086220264434816
20th percentile: 2.1260162353515626
30th percentile: 2.166644024848938
40th percentile: 2.205633926391602
50th percentile: 2.234626531600952
60th percentile: 2.328935241699219
70th percentile: 2.3877302169799806
80th percentile: 2.4505459308624267
90th percentile: 2.604385423660278
95th percentile: 2.6559148907661436
99th percentile: 2.7269501566886905
mean time: 2.3004580974578857
Pipeline stage StressChecker completed in 391.63s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.85s
Shutdown handler de-registered
chaiml-pony-d3b-mv1-win_84391_v4 status is now deployed due to DeploymentManager action
chaiml-pony-d3b-mv1-win_84391_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3b-mv1-win_84391_v4 status is now torndown due to DeploymentManager action