developer_uid: zonemercy
submission_id: chaiml-pony-d3a-mv1-son_96936_v3
model_name: chaiml-pony-d3a-mv1-son_96936_v3
model_group: ChaiML/pony-d3a-mv1-sonn
status: torndown
timestamp: 2026-03-28T17:21:12+00:00
num_battles: 11855
num_wins: 6235
celo_rating: 8468.98
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep1g8
model_architecture: Qwen3_5MoeForConditionalGeneration
model_num_parameters: 33753909248.0
best_of: 16
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3a-mv1-son_96936_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep1g8
model_size: 34B
ranking_group: single
us_pacific_date: 2026-03-25
win_ratio: 0.5259384226064951
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 1.5, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '</s>', '<|assistant|>', '####', '<|user|>'], 'max_input_tokens': 2048, 'best_of': 16, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3a-mv1-son-96936-v3-uploader
Waiting for job on chaiml-pony-d3a-mv1-son-96936-v3-uploader to finish
chaiml-pony-d3a-mv1-son-96936-v3-uploader: Using quantization_mode: none
chaiml-pony-d3a-mv1-son-96936-v3-uploader: Downloading snapshot of ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep1g8...
chaiml-pony-d3a-mv1-son-96936-v3-uploader: Downloaded in 24.947s
2026-03-25T14:42:33.256650+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
chaiml-pony-d3a-mv1-son-96936-v3-uploader: Processed model ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep1g8 in 52.038s
chaiml-pony-d3a-mv1-son-96936-v3-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3a-mv1-son-96936-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-96936-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3a-mv1-son-96936-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3a-mv1-son-96936-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3a-mv1-son-96936-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-96936-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3a-mv1-son-96936-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-96936-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3a-mv1-son-96936-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-96936-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3a-mv1-son-96936-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-96936-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3a-mv1-son-96936-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3a-mv1-son-96936-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3a-mv1-son-96936-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3a-mv1-son-96936-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3a-mv1-son-96936-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3a-mv1-son-96936-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/special_tokens_map.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/README.md
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/processor_config.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/preprocessor_config.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/generation_config.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/config.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/added_tokens.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/args.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/tokenizer_config.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/.gitattributes
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/chat_template.jinja
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/merges.txt
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model.safetensors.index.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/vocab.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/tokenizer.json
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00016-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00016-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00013-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00013-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00001-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00001-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00008-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00008-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00005-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00005-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00011-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00011-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00002-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00002-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00014-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00014-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00009-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00009-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00003-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00003-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00006-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00006-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00012-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00012-of-00016.safetensors
chaiml-pony-d3a-mv1-son-96936-v3-uploader: cp /dev/shm/model_output/model-00015-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-96936-v3/default/model-00015-of-00016.safetensors
Job chaiml-pony-d3a-mv1-son-96936-v3-uploader completed after 83.34s with status: succeeded
Stopping job with name chaiml-pony-d3a-mv1-son-96936-v3-uploader
Pipeline stage VLLMUploader completed in 84.02s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.80s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3a-mv1-son-96936-v3
Waiting for inference service chaiml-pony-d3a-mv1-son-96936-v3 to be ready
2026-03-25T14:43:33.395702+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
2026-03-25T14:44:33.850872+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
2026-03-25T14:45:34.039056+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
Inference service chaiml-pony-d3a-mv1-son-96936-v3 ready after 200.2677502632141s
Pipeline stage VLLMDeployer completed in 200.77s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-25T14:46:34.159995+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T14:47:34.256233+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 16.431105375289917s
Received healthy response to inference request in 2.0784270763397217s
Received healthy response to inference request in 6.455505847930908s
Received healthy response to inference request in 2.844815254211426s
Received healthy response to inference request in 2.4307868480682373s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T14:48:34.346596+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
Received healthy response to inference request in 1.4485061168670654s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.275709390640259s
Received healthy response to inference request in 1.5131895542144775s
Received healthy response to inference request in 1.4193115234375s
2026-03-25T14:49:34.437006+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 6.6025471687316895s
Received healthy response to inference request in 2.39394211769104s
Received healthy response to inference request in 1.8501732349395752s
Received healthy response to inference request in 1.458742380142212s
Received healthy response to inference request in 1.8248190879821777s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.6050167083740234s
2026-03-25T14:50:34.529307+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
Received healthy response to inference request in 18.177587747573853s
Received healthy response to inference request in 1.5126993656158447s
Received healthy response to inference request in 1.6541695594787598s
Received healthy response to inference request in 2.641261100769043s
Received healthy response to inference request in 1.4933347702026367s
Received healthy response to inference request in 1.6633992195129395s
30 requests
9 failed requests
5th percentile: 1.4531124353408813
10th percentile: 1.4898755311965943
20th percentile: 1.5866512775421144
30th percentile: 1.7763931274414062
40th percentile: 2.196796464920044
50th percentile: 2.53602397441864
60th percentile: 6.51432237625122
70th percentile: 18.756487560272213
80th percentile: 20.11640453338623
90th percentile: 20.126597738265993
95th percentile: 20.137640988826753
99th percentile: 20.522481191158295
mean time: 8.714326588312785
%s, retrying in %s seconds...
Received healthy response to inference request in 1.6578130722045898s
Received healthy response to inference request in 1.8634088039398193s
Received healthy response to inference request in 1.5699889659881592s
Received healthy response to inference request in 1.483506202697754s
Received healthy response to inference request in 1.376661777496338s
Received healthy response to inference request in 1.5571973323822021s
Received healthy response to inference request in 1.4385640621185303s
Received healthy response to inference request in 1.4786739349365234s
Received healthy response to inference request in 1.6042771339416504s
Received healthy response to inference request in 2.0174052715301514s
Received healthy response to inference request in 1.3758339881896973s
Received healthy response to inference request in 1.846916913986206s
Received healthy response to inference request in 1.500525951385498s
Received healthy response to inference request in 1.483098030090332s
Received healthy response to inference request in 1.5880835056304932s
Received healthy response to inference request in 1.4699418544769287s
Received healthy response to inference request in 1.5477900505065918s
Received healthy response to inference request in 1.9015922546386719s
Received healthy response to inference request in 1.7741081714630127s
Received healthy response to inference request in 1.5582525730133057s
Received healthy response to inference request in 1.5962615013122559s
Received healthy response to inference request in 1.8224620819091797s
Received healthy response to inference request in 1.7026753425598145s
Received healthy response to inference request in 1.5465853214263916s
Received healthy response to inference request in 1.4925124645233154s
Received healthy response to inference request in 1.4529743194580078s
Received healthy response to inference request in 1.903940200805664s
Received healthy response to inference request in 1.448676586151123s
2026-03-25T14:51:34.626948+00:00 monitor updated for chaiml-pony-d3a-mv1-son_96936_v3
Received healthy response to inference request in 1.6893584728240967s
Received healthy response to inference request in 2.063185453414917s
30 requests
0 failed requests
5th percentile: 1.4045178055763246
10th percentile: 1.4476653337478638
20th percentile: 1.4769275188446045
30th percentile: 1.489810585975647
40th percentile: 1.5473081588745117
50th percentile: 1.5641207695007324
60th percentile: 1.5994677543640137
70th percentile: 1.693353533744812
80th percentile: 1.827353048324585
90th percentile: 1.901827049255371
95th percentile: 1.9663459897041318
99th percentile: 2.049909200668335
mean time: 1.627075719833374
Pipeline stage StressChecker completed in 317.20s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.28s
Shutdown handler de-registered
chaiml-pony-d3a-mv1-son_96936_v3 status is now deployed due to DeploymentManager action
chaiml-pony-d3a-mv1-son_96936_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3a-mv1-son_96936_v3 status is now torndown due to DeploymentManager action