chaiml-pony-d3a-mv1-son_75599

developer_uid: zonemercy

submission_id: chaiml-pony-d3a-mv1-son_75599_v5

model_name: chaiml-pony-d3a-mv1-son_75599_v5

model_group: ChaiML/pony-d3a-mv1-sonn

status: torndown

timestamp: 2026-03-31T18:21:56+00:00

num_battles: 11652

num_wins: 6086

celo_rating: 1307.61

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep2g8

model_architecture: Qwen3_5MoeForConditionalGeneration

model_num_parameters: 33753909248.0

best_of: 16

max_input_tokens: 2048

max_output_tokens: 80

reward_model: default

display_name: chaiml-pony-d3a-mv1-son_75599_v5

ineligible_reason: max_output_tokens!=64

is_internal_developer: True

language_model: ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep2g8

model_size: 34B

ranking_group: single

us_pacific_date: 2026-03-28

win_ratio: 0.5223137658771027

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '####', '<|assistant|>', '<|user|>', '</s>'], 'max_input_tokens': 2048, 'best_of': 16, 'max_output_tokens': 80}

formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3a-mv1-son-75599-v5-uploader
Waiting for job on chaiml-pony-d3a-mv1-son-75599-v5-uploader to finish
chaiml-pony-d3a-mv1-son-75599-v5-uploader: Using quantization_mode: fp8
chaiml-pony-d3a-mv1-son-75599-v5-uploader: Checking if ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-pony-d3a-mv1-son-75599-v5-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-d3a-mv1-son-75599-v5-uploader: Downloading snapshot of ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep2g8-FP8...
2026-03-28T14:49:28.776719+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
chaiml-pony-d3a-mv1-son-75599-v5-uploader: Downloaded in 34.980s
chaiml-pony-d3a-mv1-son-75599-v5-uploader: Processed model ChaiML/pony-d3a-mv1-sonnetwintop2-q35b-lr5e6ep2g8 in 37.468s
chaiml-pony-d3a-mv1-son-75599-v5-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3a-mv1-son-75599-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-75599-v5-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3a-mv1-son-75599-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3a-mv1-son-75599-v5-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3a-mv1-son-75599-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-75599-v5-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3a-mv1-son-75599-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-75599-v5-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3a-mv1-son-75599-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-75599-v5-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3a-mv1-son-75599-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-son-75599-v5-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3a-mv1-son-75599-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3a-mv1-son-75599-v5-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3a-mv1-son-75599-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3a-mv1-son-75599-v5-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3a-mv1-son-75599-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3a-mv1-son-75599-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default
chaiml-pony-d3a-mv1-son-75599-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default/.gitattributes
chaiml-pony-d3a-mv1-son-75599-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default/config.json
chaiml-pony-d3a-mv1-son-75599-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default/tokenizer_config.json
chaiml-pony-d3a-mv1-son-75599-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default/generation_config.json
chaiml-pony-d3a-mv1-son-75599-v5-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default/recipe.yaml
chaiml-pony-d3a-mv1-son-75599-v5-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default/chat_template.jinja
chaiml-pony-d3a-mv1-son-75599-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default/tokenizer.json
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-28T14:50:28.881141+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
Failed to get request counts for guanaco-submitter. Falling back to default
chaiml-pony-d3a-mv1-son-75599-v5-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-son-75599-v5/default/model.safetensors
Job chaiml-pony-d3a-mv1-son-75599-v5-uploader completed after 144.58s with status: succeeded
Stopping job with name chaiml-pony-d3a-mv1-son-75599-v5-uploader
Pipeline stage VLLMUploader completed in 145.10s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.88s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3a-mv1-son-75599-v5
Waiting for inference service chaiml-pony-d3a-mv1-son-75599-v5 to be ready
2026-03-28T14:51:28.975320+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
2026-03-28T14:52:29.177117+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
2026-03-28T14:53:29.549858+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
Inference service chaiml-pony-d3a-mv1-son-75599-v5 ready after 180.97058296203613s
Pipeline stage VLLMDeployer completed in 182.40s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T14:54:29.776474+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 10.607706546783447s
2026-03-28T14:55:30.006733+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 5.767673015594482s
Received healthy response to inference request in 3.7505836486816406s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.3033335208892822s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T14:56:30.186126+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.259577989578247s
Received healthy response to inference request in 5.816188812255859s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.668899774551392s
Received healthy response to inference request in 4.572282791137695s
2026-03-28T14:57:30.285619+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.4514474868774414s
Received healthy response to inference request in 1.6580727100372314s
Retrying (%r) after connection broken by '%r': %s
Received healthy response to inference request in 20.378191232681274s
Received healthy response to inference request in 1.4261870384216309s
Received healthy response to inference request in 1.2900991439819336s
Received healthy response to inference request in 2.781991481781006s
Received healthy response to inference request in 1.6136841773986816s
Received healthy response to inference request in 1.287445068359375s
Received healthy response to inference request in 4.092505693435669s
Received healthy response to inference request in 1.3202488422393799s
Received healthy response to inference request in 1.3783016204833984s
Received healthy response to inference request in 1.6865830421447754s
Received healthy response to inference request in 1.512275218963623s
30 requests
9 failed requests
5th percentile: 1.2886394023895265
10th percentile: 1.3020100831985473
20th percentile: 1.4166099548339843
30th percentile: 1.583261489868164
40th percentile: 2.3438281059265154
50th percentile: 4.332394242286682
60th percentile: 5.787079334259033
70th percentile: 20.121564435958863
80th percentile: 20.141918563842772
90th percentile: 20.20244369506836
95th percentile: 20.361664211750032
99th percentile: 20.77055900335312
mean time: 8.730512404441834
%s, retrying in %s seconds...
Received healthy response to inference request in 1.2607462406158447s
Received healthy response to inference request in 1.513728141784668s
2026-03-28T14:58:30.401673+00:00 monitor updated for chaiml-pony-d3a-mv1-son_75599_v5
Received healthy response to inference request in 1.3862385749816895s
Received healthy response to inference request in 1.1480906009674072s
Received healthy response to inference request in 1.1948449611663818s
Received healthy response to inference request in 1.2442514896392822s
Received healthy response to inference request in 1.3710689544677734s
Received healthy response to inference request in 1.7335805892944336s
Received healthy response to inference request in 1.3028976917266846s
Received healthy response to inference request in 1.3446106910705566s
Received healthy response to inference request in 1.313819408416748s
Received healthy response to inference request in 1.334183931350708s
Received healthy response to inference request in 1.2635133266448975s
Received healthy response to inference request in 1.458237886428833s
Received healthy response to inference request in 1.3037526607513428s
Received healthy response to inference request in 1.26444411277771s
Received healthy response to inference request in 1.3427565097808838s
Received healthy response to inference request in 1.7731282711029053s
Received healthy response to inference request in 1.2631332874298096s
Received healthy response to inference request in 1.49350905418396s
Received healthy response to inference request in 1.3148748874664307s
Received healthy response to inference request in 1.3675141334533691s
Received healthy response to inference request in 1.2107524871826172s
Received healthy response to inference request in 1.3231406211853027s
Received healthy response to inference request in 1.2858350276947021s
Received healthy response to inference request in 1.332007884979248s
Received healthy response to inference request in 1.3379478454589844s
Received healthy response to inference request in 1.357347011566162s
Received healthy response to inference request in 1.8089966773986816s
Received healthy response to inference request in 1.3047447204589844s
30 requests
0 failed requests
5th percentile: 1.2020033478736878
10th percentile: 1.2409015893936157
20th percentile: 1.26343731880188
30th percentile: 1.2977788925170899
40th percentile: 1.3101895332336426
50th percentile: 1.3275742530822754
60th percentile: 1.339871311187744
70th percentile: 1.3603971481323243
80th percentile: 1.4006384372711185
90th percentile: 1.535713386535645
95th percentile: 1.755331814289093
99th percentile: 1.7985948395729066
mean time: 1.3651232560475668
Pipeline stage StressChecker completed in 309.94s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.73s
Shutdown handler de-registered
chaiml-pony-d3a-mv1-son_75599_v5 status is now deployed due to DeploymentManager action
chaiml-pony-d3a-mv1-son_75599_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3a-mv1-son_75599_v5 status is now torndown due to DeploymentManager action