developer_uid: chai_backend_admin
submission_id: chaiml-pony-d3b-mv1-top2_9386_v9
model_name: chaiml-pony-d3b-mv1-top2_9386_v9
model_group: ChaiML/pony-d3b-mv1-top2
status: torndown
timestamp: 2026-03-31T23:21:44+00:00
num_battles: 10845
num_wins: 5512
celo_rating: 1306.08
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8
model_architecture: Qwen3_5MoeForConditionalGeneration
model_num_parameters: 33753909248.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3b-mv1-top2_9386_v9
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8
model_size: 34B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5082526509912402
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.8, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '####', '<|assistant|>', '<|im_end|>', '<|user|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3b-mv1-top2-9386-v9-uploader
Waiting for job on chaiml-pony-d3b-mv1-top2-9386-v9-uploader to finish
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: Using quantization_mode: fp8
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: Checking if ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: Downloading snapshot of ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8-FP8...
2026-03-28T20:14:40.473450+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: Downloaded in 34.950s
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: Processed model ChaiML/pony-d3b-mv1-top2-q35b-lr5e6ep2g8 in 37.788s
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v9-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3b-mv1-top2-9386-v9-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v9-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v9-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v9-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3b-mv1-top2-9386-v9-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3b-mv1-top2-9386-v9-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3b-mv1-top2-9386-v9-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default/.gitattributes
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default/config.json
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default/recipe.yaml
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default/chat_template.jinja
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default/generation_config.json
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default/tokenizer_config.json
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default/tokenizer.json
2026-03-28T20:15:40.565926+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
chaiml-pony-d3b-mv1-top2-9386-v9-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-d3b-mv1-top2-9386-v9/default/model.safetensors
Job chaiml-pony-d3b-mv1-top2-9386-v9-uploader completed after 143.55s with status: succeeded
Stopping job with name chaiml-pony-d3b-mv1-top2-9386-v9-uploader
Pipeline stage VLLMUploader completed in 144.00s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.33s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3b-mv1-top2-9386-v9
Waiting for inference service chaiml-pony-d3b-mv1-top2-9386-v9 to be ready
2026-03-28T20:16:40.666782+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
2026-03-28T20:17:40.756231+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
2026-03-28T20:18:40.848102+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
Inference service chaiml-pony-d3b-mv1-top2-9386-v9 ready after 210.44436597824097s
Pipeline stage VLLMDeployer completed in 210.93s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-28T20:19:40.950189+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Failed to get request counts for guanaco-submitter. Falling back to default
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T20:20:41.047272+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.160443067550659s
Received healthy response to inference request in 3.726484775543213s
Received healthy response to inference request in 2.3561415672302246s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.4596734046936035s
Received healthy response to inference request in 2.005391836166382s
Received healthy response to inference request in 1.8099040985107422s
2026-03-28T20:21:41.147822+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.8017916679382324s
Received healthy response to inference request in 2.1910157203674316s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.2217233180999756s
Received healthy response to inference request in 1.7553162574768066s
Received healthy response to inference request in 2.2571778297424316s
Received healthy response to inference request in 1.6654882431030273s
Received healthy response to inference request in 1.894824504852295s
2026-03-28T20:22:41.254113+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.2945919036865234s
Received healthy response to inference request in 1.637805461883545s
Received healthy response to inference request in 1.9203150272369385s
Received healthy response to inference request in 1.7024321556091309s
Received healthy response to inference request in 1.9135136604309082s
Received healthy response to inference request in 1.817478895187378s
Received healthy response to inference request in 1.6425251960754395s
Received healthy response to inference request in 1.6933372020721436s
Received healthy response to inference request in 2.3707950115203857s
30 requests
8 failed requests
5th percentile: 1.652858567237854
10th percentile: 1.690552306175232
20th percentile: 1.7989865303039552
30th percentile: 1.9079069137573241
40th percentile: 2.116766166687012
50th percentile: 2.306659698486328
60th percentile: 2.7399812698364245
70th percentile: 3.7490768432617188
80th percentile: 20.113595485687256
90th percentile: 20.129631423950194
95th percentile: 20.15760360956192
99th percentile: 20.433561992645263
mean time: 7.024299772580465
%s, retrying in %s seconds...
Received healthy response to inference request in 1.669163465499878s
Received healthy response to inference request in 1.5980873107910156s
Received healthy response to inference request in 1.618807077407837s
Received healthy response to inference request in 1.6668975353240967s
Received healthy response to inference request in 1.6455042362213135s
Received healthy response to inference request in 1.6503994464874268s
Received healthy response to inference request in 2.1427464485168457s
Received healthy response to inference request in 2.0158538818359375s
Received healthy response to inference request in 1.6565334796905518s
Received healthy response to inference request in 1.7369287014007568s
Received healthy response to inference request in 1.7103636264801025s
Received healthy response to inference request in 1.782024621963501s
Received healthy response to inference request in 1.764256238937378s
Received healthy response to inference request in 1.6888427734375s
Received healthy response to inference request in 1.9632234573364258s
Received healthy response to inference request in 1.8392698764801025s
2026-03-28T20:23:41.410107+00:00 monitor updated for chaiml-pony-d3b-mv1-top2_9386_v9
Received healthy response to inference request in 1.6341288089752197s
Received healthy response to inference request in 2.0799925327301025s
Received healthy response to inference request in 2.278641700744629s
Received healthy response to inference request in 1.704789161682129s
Received healthy response to inference request in 1.7516098022460938s
Received healthy response to inference request in 1.8903367519378662s
Received healthy response to inference request in 1.6871685981750488s
Received healthy response to inference request in 1.6717934608459473s
Received healthy response to inference request in 1.6807069778442383s
Received healthy response to inference request in 1.8427343368530273s
Received healthy response to inference request in 1.72883939743042s
Received healthy response to inference request in 1.6392590999603271s
Received healthy response to inference request in 1.6290132999420166s
Received healthy response to inference request in 1.809086561203003s
30 requests
0 failed requests
5th percentile: 1.6233998775482177
10th percentile: 1.6336172580718995
20th percentile: 1.649420404434204
30th percentile: 1.6684836864471435
40th percentile: 1.6845839500427247
50th percentile: 1.7075763940811157
60th percentile: 1.7428011417388916
70th percentile: 1.7901432037353515
80th percentile: 1.8522548198699953
90th percentile: 2.0222677469253543
95th percentile: 2.114507186412811
99th percentile: 2.239232077598572
mean time: 1.7725667556126912
Pipeline stage StressChecker completed in 268.66s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.09s
Shutdown handler de-registered
chaiml-pony-d3b-mv1-top2_9386_v9 status is now deployed due to DeploymentManager action
chaiml-pony-d3b-mv1-top2_9386_v9 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3b-mv1-top2_9386_v9 status is now torndown due to DeploymentManager action