chaiml-pony-v3-q27b-lr5_22882

developer_uid: zonemercy
submission_id: chaiml-pony-v3-q27b-lr5_22882_v6
model_name: chaiml-pony-v3-q27b-lr5_22882_v6
model_group: ChaiML/pony-v3-q27b-lr5e
status: inactive
timestamp: 2026-03-29T12:17:10+00:00
num_battles: 10510
num_wins: 5482
celo_rating: 1308.59
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v3-q27b-lr5e6ep1g8
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v3-q27b-lr5_22882_v6
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v3-q27b-lr5e6ep1g8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-29
win_ratio: 0.5215984776403425
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|user|>', '</s>', '<|im_end|>', '<|assistant|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v3-q27b-lr5-22882-v6-uploader
Waiting for job on chaiml-pony-v3-q27b-lr5-22882-v6-uploader to finish
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: Using quantization_mode: fp8
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: Checking if ChaiML/pony-v3-q27b-lr5e6ep1g8-FP8 already exists in ChaiML
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: Downloading snapshot of ChaiML/pony-v3-q27b-lr5e6ep1g8-FP8...
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: Downloaded in 32.575s
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: Processed model ChaiML/pony-v3-q27b-lr5e6ep1g8 in 35.406s
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v6-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v3-q27b-lr5-22882-v6-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v6-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v6-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v6-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3-q27b-lr5-22882-v6-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v3-q27b-lr5-22882-v6-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v3-q27b-lr5-22882-v6-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default/generation_config.json
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default/tokenizer_config.json
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default/recipe.yaml
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default/config.json
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default/chat_template.jinja
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default/.gitattributes
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default/tokenizer.json
2026-03-29T10:03:38.469540+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
chaiml-pony-v3-q27b-lr5-22882-v6-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-v3-q27b-lr5-22882-v6/default/model.safetensors
Job chaiml-pony-v3-q27b-lr5-22882-v6-uploader completed after 112.77s with status: succeeded
Stopping job with name chaiml-pony-v3-q27b-lr5-22882-v6-uploader
Pipeline stage VLLMUploader completed in 113.81s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.54s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v3-q27b-lr5-22882-v6
Waiting for inference service chaiml-pony-v3-q27b-lr5-22882-v6 to be ready
2026-03-29T10:04:38.569186+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
2026-03-29T10:05:38.679302+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
2026-03-29T10:06:38.781091+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
Inference service chaiml-pony-v3-q27b-lr5-22882-v6 ready after 180.5775442123413s
Pipeline stage VLLMDeployer completed in 181.03s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-29T10:07:38.894317+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-29T10:08:39.010908+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
Failed to get response for submission chaiml-q235b-judge-dpo-_74524_v1: HTTPConnectionPool(host='chaiml-q235b-judge-dpo-74524-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 11.915760278701782s
2026-03-29T10:09:39.151935+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.9757676124572754s
Received healthy response to inference request in 4.242753505706787s
Received healthy response to inference request in 4.426192998886108s
2026-03-29T10:10:39.263685+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
Received healthy response to inference request in 13.779374599456787s
Received healthy response to inference request in 1.9579505920410156s
Received healthy response to inference request in 4.242834091186523s
Received healthy response to inference request in 1.8670928478240967s
Received healthy response to inference request in 1.9501309394836426s
Received healthy response to inference request in 1.9039771556854248s
Received healthy response to inference request in 1.9431002140045166s
Received healthy response to inference request in 2.006819009780884s
Received healthy response to inference request in 2.026581287384033s
Received healthy response to inference request in 2.0131194591522217s
Received healthy response to inference request in 1.9376065731048584s
Received healthy response to inference request in 2.211578845977783s
Received healthy response to inference request in 2.1596179008483887s
Received healthy response to inference request in 2.6374573707580566s
Received healthy response to inference request in 2.0085017681121826s
Received healthy response to inference request in 2.0921881198883057s
Received healthy response to inference request in 2.0145182609558105s
Failed to get response for submission chaiml-q235b-judge-dpo-_74524_v1: HTTPConnectionPool(host='chaiml-q235b-judge-dpo-74524-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received healthy response to inference request in 2.03058123588562s
Received healthy response to inference request in 2.138068199157715s
30 requests
7 failed requests
5th percentile: 1.91911039352417
10th percentile: 1.9425508499145507
20th percentile: 1.9722042083740234
30th percentile: 2.01173415184021
40th percentile: 2.028981256484985
50th percentile: 2.1488430500030518
60th percentile: 3.2795758247375466
70th percentile: 6.67306318283079
80th percentile: 20.121103715896606
90th percentile: 20.176250457763672
95th percentile: 20.277124416828155
99th percentile: 20.610187425613404
mean time: 7.244307406743368
%s, retrying in %s seconds...
Received healthy response to inference request in 1.958970069885254s
Received healthy response to inference request in 1.7570695877075195s
Received healthy response to inference request in 1.755277156829834s
Received healthy response to inference request in 2.1330745220184326s
Received healthy response to inference request in 1.9211926460266113s
Received healthy response to inference request in 1.9303598403930664s
2026-03-29T10:11:39.497620+00:00 monitor updated for chaiml-pony-v3-q27b-lr5_22882_v6
Received healthy response to inference request in 1.8100836277008057s
Received healthy response to inference request in 1.8257880210876465s
Received healthy response to inference request in 1.7608988285064697s
Received healthy response to inference request in 1.8770477771759033s
Received healthy response to inference request in 1.9136462211608887s
Received healthy response to inference request in 1.9015226364135742s
Received healthy response to inference request in 1.867363452911377s
Received healthy response to inference request in 2.000056028366089s
Received healthy response to inference request in 1.9072167873382568s
Received healthy response to inference request in 2.0846738815307617s
Received healthy response to inference request in 1.936704397201538s
Received healthy response to inference request in 2.0126779079437256s
Received healthy response to inference request in 1.9299960136413574s
Received healthy response to inference request in 1.9202005863189697s
Received healthy response to inference request in 1.9856431484222412s
Received healthy response to inference request in 1.9391157627105713s
Received healthy response to inference request in 1.9425904750823975s
Received healthy response to inference request in 2.105086088180542s
Received healthy response to inference request in 1.98368501663208s
Received healthy response to inference request in 2.356177806854248s
Received healthy response to inference request in 2.0333456993103027s
Received healthy response to inference request in 2.107452392578125s
Received healthy response to inference request in 2.1750662326812744s
Received healthy response to inference request in 2.127835512161255s
30 requests
0 failed requests
5th percentile: 1.7587927460670472
10th percentile: 1.8051651477813722
20th percentile: 1.875110912322998
30th percentile: 1.9117173910140992
40th percentile: 1.926474666595459
50th percentile: 1.9379100799560547
60th percentile: 1.9688560485839843
70th percentile: 2.00384259223938
80th percentile: 2.0887563228607178
90th percentile: 2.1283594131469727
95th percentile: 2.1561699628829953
99th percentile: 2.303655450344086
mean time: 1.965327270825704
Pipeline stage StressChecker completed in 294.79s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.21s
Shutdown handler de-registered
chaiml-pony-v3-q27b-lr5_22882_v6 status is now deployed due to DeploymentManager action
chaiml-pony-v3-q27b-lr5_22882_v6 status is now inactive due to auto deactivation removed underperforming models