developer_uid: zonemercy
submission_id: chaiml-pony-d3b-mv1-top2_9386_v4
model_name: chaiml-pony-d3b-mv1-top2_9386_v4
model_group: ChaiML/pony-d3b-mv1-top2
status: torndown
timestamp: 2026-03-31T03:51:02+00:00
num_battles: 11044
num_wins: 5632
celo_rating: 1299.28
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8
model_architecture: Qwen3_5MoeForConditionalGeneration
model_num_parameters: 33753909248.0
best_of: 16
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3b-mv1-top2_9386_v4
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8
model_size: 34B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5099601593625498
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.8, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '</s>', '####', '<|assistant|>', '<|user|>'], 'max_input_tokens': 2048, 'best_of': 16, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3b-mv1-top2-9386-v4-uploader
Waiting for job on chaiml-pony-d3b-mv1-top2-9386-v4-uploader to finish
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: Using quantization_mode: fp8
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: Checking if ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: Downloading snapshot of ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8-FP8...
2026-03-28T01:24:03.936000+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: Downloaded in 37.247s
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: Processed model ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8 in 39.749s
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3b-mv1-top2-9386-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3b-mv1-top2-9386-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3b-mv1-top2-9386-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default/recipe.yaml
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default/.gitattributes
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default/tokenizer_config.json
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default/generation_config.json
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default/chat_template.jinja
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default/config.json
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default/tokenizer.json
2026-03-28T01:25:04.024109+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
chaiml-pony-d3b-mv1-top2-9386-v4-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v4/default/model.safetensors
Job chaiml-pony-d3b-mv1-top2-9386-v4-uploader completed after 132.83s with status: succeeded
Stopping job with name chaiml-pony-d3b-mv1-top2-9386-v4-uploader
Pipeline stage VLLMUploader completed in 133.39s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.77s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3b-mv1-top2-9386-v4
Waiting for inference service chaiml-pony-d3b-mv1-top2-9386-v4 to be ready
2026-03-28T01:26:04.109114+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
2026-03-28T01:27:04.200634+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
2026-03-28T01:28:04.314739+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
Inference service chaiml-pony-d3b-mv1-top2-9386-v4 ready after 190.7765097618103s
Pipeline stage VLLMDeployer completed in 191.50s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T01:29:04.406135+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 12.93438196182251s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T01:30:04.504315+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 5.728819131851196s
Received healthy response to inference request in 2.547060489654541s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.734290361404419s
Received healthy response to inference request in 5.758074522018433s
2026-03-28T01:31:04.611986+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
{"detail":"('http://chaiml-pony-d3b-mv1-top2-9386-v4-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination')"}
Received unhealthy response to inference request!
Received healthy response to inference request in 2.586747646331787s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.3703432083129883s
Received healthy response to inference request in 2.080008029937744s
Received healthy response to inference request in 2.597524642944336s
Received healthy response to inference request in 2.2147083282470703s
Received healthy response to inference request in 2.141000986099243s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T01:32:04.711760+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
Received healthy response to inference request in 2.091151237487793s
Received healthy response to inference request in 2.1342108249664307s
Received healthy response to inference request in 2.678215742111206s
Received healthy response to inference request in 2.700171947479248s
Received healthy response to inference request in 2.4700117111206055s
Received healthy response to inference request in 2.2368521690368652s
Received healthy response to inference request in 2.2079696655273438s
Received healthy response to inference request in 2.623527765274048s
Received healthy response to inference request in 2.1618285179138184s
Received healthy response to inference request in 2.1031484603881836s
Received healthy response to inference request in 2.165931224822998s
30 requests
8 failed requests
5th percentile: 2.0965499877929688
10th percentile: 2.131104588508606
20th percentile: 2.165110683441162
30th percentile: 2.2302090167999267
40th percentile: 2.5162409782409667
50th percentile: 2.610526204109192
60th percentile: 2.7138193130493162
70th percentile: 7.910966753959635
80th percentile: 20.117267990112303
90th percentile: 20.15811655521393
95th percentile: 20.265987813472748
99th percentile: 20.54363250494003
mean time: 7.61214477221171
%s, retrying in %s seconds...
Received healthy response to inference request in 2.083310127258301s
Received healthy response to inference request in 2.086311101913452s
Received healthy response to inference request in 2.0751290321350098s
Received healthy response to inference request in 2.0780434608459473s
Received healthy response to inference request in 2.0096890926361084s
Received healthy response to inference request in 2.316004753112793s
Received healthy response to inference request in 2.1509180068969727s
Received healthy response to inference request in 2.3344290256500244s
Received healthy response to inference request in 2.1505284309387207s
Received healthy response to inference request in 2.086188316345215s
Received healthy response to inference request in 2.6363728046417236s
Received healthy response to inference request in 2.563891887664795s
Received healthy response to inference request in 2.1810660362243652s
Received healthy response to inference request in 2.2539470195770264s
2026-03-28T01:33:04.860431+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v4
Received healthy response to inference request in 2.311884641647339s
Received healthy response to inference request in 2.2146527767181396s
Received healthy response to inference request in 2.1135942935943604s
Received healthy response to inference request in 2.19681715965271s
Received healthy response to inference request in 2.2429559230804443s
Received healthy response to inference request in 2.184697151184082s
Received healthy response to inference request in 2.620570182800293s
Received healthy response to inference request in 2.339083194732666s
Received healthy response to inference request in 2.1702427864074707s
Received healthy response to inference request in 2.5480141639709473s
Received healthy response to inference request in 2.233166456222534s
Received healthy response to inference request in 2.0873630046844482s
Received healthy response to inference request in 2.1037070751190186s
Received healthy response to inference request in 2.4260671138763428s
Received healthy response to inference request in 2.0873069763183594s
Received healthy response to inference request in 2.187424898147583s
30 requests
0 failed requests
5th percentile: 2.076440525054932
10th percentile: 2.0827834606170654
20th percentile: 2.087107801437378
30th percentile: 2.110628128051758
40th percentile: 2.1625128746032716
50th percentile: 2.1860610246658325
60th percentile: 2.2220582485198976
70th percentile: 2.2713283061981198
80th percentile: 2.3353598594665526
90th percentile: 2.549601936340332
95th percentile: 2.5950649499893186
99th percentile: 2.6317900443077087
mean time: 2.2357792297999066
Pipeline stage StressChecker completed in 309.63s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.50s
Shutdown handler de-registered
chaiml-pony-d3b-mv1-top2_9386_v4 status is now deployed due to DeploymentManager action
chaiml-pony-d3b-mv1-top2_9386_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3b-mv1-top2_9386_v4 status is now torndown due to DeploymentManager action