developer_uid: zonemercy
submission_id: chaiml-pony-d3a-mv1-plc_30375_v3
model_name: chaiml-pony-d3a-mv1-plc_30375_v3
model_group: ChaiML/pony-d3a-mv1-plc-
status: torndown
timestamp: 2026-03-28T17:21:14+00:00
num_battles: 10536
num_wins: 5546
celo_rating: 8468.98
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep1g8
model_architecture: Qwen3_5MoeForConditionalGeneration
model_num_parameters: 33753909248.0
best_of: 16
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3a-mv1-plc_30375_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep1g8
model_size: 34B
ranking_group: single
us_pacific_date: 2026-03-25
win_ratio: 0.5263857251328777
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 1.5, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '</s>', '<|user|>', '####', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 16, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3a-mv1-plc-30375-v3-uploader
Waiting for job on chaiml-pony-d3a-mv1-plc-30375-v3-uploader to finish
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: Using quantization_mode: none
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: Downloading snapshot of ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep1g8...
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: Downloaded in 27.838s
2026-03-25T14:56:30.858922+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: Processed model ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep1g8 in 57.753s
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-30375-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3a-mv1-plc-30375-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-30375-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-30375-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-30375-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-30375-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3a-mv1-plc-30375-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3a-mv1-plc-30375-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/README.md
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/special_tokens_map.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/preprocessor_config.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model.safetensors.index.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/vocab.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/processor_config.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/.gitattributes
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/added_tokens.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/generation_config.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/tokenizer_config.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/chat_template.jinja
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/args.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/config.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/merges.txt
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/tokenizer.json
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00016-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00016-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00013-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00013-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00004-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00004-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00010-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00010-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00007-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00007-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00005-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00005-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00008-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00008-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00001-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00001-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00003-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00003-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00006-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00006-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00014-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00014-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00011-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00011-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00002-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00002-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00015-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00015-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00009-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00009-of-00016.safetensors
chaiml-pony-d3a-mv1-plc-30375-v3-uploader: cp /dev/shm/model_output/model-00012-of-00016.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-30375-v3/default/model-00012-of-00016.safetensors
Job chaiml-pony-d3a-mv1-plc-30375-v3-uploader completed after 94.23s with status: succeeded
Stopping job with name chaiml-pony-d3a-mv1-plc-30375-v3-uploader
Pipeline stage VLLMUploader completed in 94.88s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.53s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3a-mv1-plc-30375-v3
Waiting for inference service chaiml-pony-d3a-mv1-plc-30375-v3 to be ready
2026-03-25T14:57:30.944433+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
Failed to get response for submission chaiml-pony-d3a-mv1-son_96936_v1: ('http://chaiml-pony-d3a-mv1-son-96936-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
2026-03-25T14:58:31.034304+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
2026-03-25T14:59:31.123472+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
Inference service chaiml-pony-d3a-mv1-plc-30375-v3 ready after 190.28030037879944s
Pipeline stage VLLMDeployer completed in 190.94s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-25T15:00:31.215270+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T15:01:31.306842+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-25T15:02:31.798738+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 17.42949652671814s
Received healthy response to inference request in 2.1242432594299316s
Received healthy response to inference request in 7.227001667022705s
Received healthy response to inference request in 6.381949424743652s
Received healthy response to inference request in 1.5506277084350586s
Received healthy response to inference request in 2.2323896884918213s
Received healthy response to inference request in 2.093411922454834s
2026-03-25T15:03:32.000255+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.7106890678405762s
Received healthy response to inference request in 1.4126713275909424s
Received healthy response to inference request in 1.4047749042510986s
Received healthy response to inference request in 1.561979055404663s
Received healthy response to inference request in 1.5474944114685059s
Received healthy response to inference request in 1.9820892810821533s
Received healthy response to inference request in 2.3873674869537354s
Received healthy response to inference request in 1.6350107192993164s
Received healthy response to inference request in 1.7501742839813232s
Received healthy response to inference request in 1.4834461212158203s
Received healthy response to inference request in 1.673088788986206s
Received healthy response to inference request in 1.4349887371063232s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.5479705333709717s
Received healthy response to inference request in 1.4492042064666748s
30 requests
9 failed requests
5th percentile: 1.4227141618728638
10th percentile: 1.4477826595306396
20th percentile: 1.5478753089904784
30th percentile: 1.6131012201309203
40th percentile: 1.7343801975250244
50th percentile: 2.108827590942383
60th percentile: 3.9852002620696965
70th percentile: 18.23453338146209
80th percentile: 20.12424030303955
90th percentile: 20.137787294387817
95th percentile: 20.15009479522705
99th percentile: 20.485905432701113
mean time: 8.12324198881785
%s, retrying in %s seconds...
2026-03-25T15:04:32.092913+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_30375_v3
Received healthy response to inference request in 9.13615608215332s
Received healthy response to inference request in 2.113368272781372s
Received healthy response to inference request in 1.3671867847442627s
Received healthy response to inference request in 1.3148493766784668s
Received healthy response to inference request in 1.8340301513671875s
Received healthy response to inference request in 2.0537683963775635s
Received healthy response to inference request in 1.4721848964691162s
Received healthy response to inference request in 1.273829460144043s
Received healthy response to inference request in 1.8546710014343262s
Received healthy response to inference request in 1.3655452728271484s
Received healthy response to inference request in 1.6966032981872559s
Received healthy response to inference request in 2.0125629901885986s
Received healthy response to inference request in 1.4450957775115967s
Received healthy response to inference request in 1.8592078685760498s
Received healthy response to inference request in 1.7530415058135986s
Received healthy response to inference request in 1.4190144538879395s
Received healthy response to inference request in 1.5113155841827393s
Received healthy response to inference request in 1.4000484943389893s
Received healthy response to inference request in 1.7611513137817383s
Received healthy response to inference request in 1.4402973651885986s
Received healthy response to inference request in 1.4864130020141602s
Received healthy response to inference request in 1.5275049209594727s
Received healthy response to inference request in 1.6214227676391602s
Received healthy response to inference request in 1.4453351497650146s
Received healthy response to inference request in 1.758591651916504s
Received healthy response to inference request in 2.2493021488189697s
Received healthy response to inference request in 1.411698341369629s
Received healthy response to inference request in 1.54439115524292s
Received healthy response to inference request in 1.4324572086334229s
Received healthy response to inference request in 1.6849596500396729s
30 requests
0 failed requests
5th percentile: 1.3376625299453735
10th percentile: 1.3670226335525513
20th percentile: 1.4175512313842773
30th percentile: 1.4436562538146973
40th percentile: 1.4807217597961426
50th percentile: 1.5359480381011963
60th percentile: 1.689617109298706
70th percentile: 1.7593595504760742
80th percentile: 1.8555783748626709
90th percentile: 2.0597283840179443
95th percentile: 2.1881319046020504
99th percentile: 7.138968441486364
mean time: 1.8748668114344278
Pipeline stage StressChecker completed in 307.99s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.60s
Shutdown handler de-registered
chaiml-pony-d3a-mv1-plc_30375_v3 status is now deployed due to DeploymentManager action
chaiml-pony-d3a-mv1-plc_30375_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3a-mv1-plc_30375_v3 status is now torndown due to DeploymentManager action