developer_uid: zonemercy
submission_id: chaiml-pony-v2-q27b-lr1_74562_v8
model_name: chaiml-pony-v2-q27b-lr1_74562_v8
model_group: ChaiML/pony-v2-q27b-lr1e
status: inactive
timestamp: 2026-03-18T08:16:45+00:00
num_battles: 10205
num_wins: 5340
celo_rating: 1309.61
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v2-q27b-lr1e4ep1r64g8
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v2-q27b-lr1_74562_v8
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v2-q27b-lr1e4ep1r64g8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-18
win_ratio: 0.5232729054385106
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-q27b-lr1-74562-v8-uploader
Waiting for job on chaiml-pony-v2-q27b-lr1-74562-v8-uploader to finish
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: Using quantization_mode: none
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: Downloading snapshot of ChaiML/pony-v2-q27b-lr1e4ep1r64g8...
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: Downloaded in 34.546s
2026-03-18T00:39:49.530532+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: Processed model ChaiML/pony-v2-q27b-lr1e4ep1r64g8 in 55.759s
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v8-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-q27b-lr1-74562-v8-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v8-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v8-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v8-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v8-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-q27b-lr1-74562-v8-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-q27b-lr1-74562-v8-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/processor_config.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/config.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/args.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/added_tokens.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/generation_config.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/merges.txt
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/.gitattributes
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/trainer_state.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/trainer_state.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/preprocessor_config.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/README.md
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/special_tokens_map.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/training_args.bin
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/tokenizer_config.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/chat_template.jinja
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/model.safetensors.index.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/tokenizer.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/vocab.json
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/model-00002-of-00002.safetensors
2026-03-18T00:40:49.618155+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
2026-03-18T00:41:49.721497+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
2026-03-18T00:42:49.821668+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
2026-03-18T00:43:49.967637+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
2026-03-18T00:44:50.090836+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
chaiml-pony-v2-q27b-lr1-74562-v8-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v8/default/model-00001-of-00002.safetensors
Job chaiml-pony-v2-q27b-lr1-74562-v8-uploader completed after 396.49s with status: succeeded
Stopping job with name chaiml-pony-v2-q27b-lr1-74562-v8-uploader
Pipeline stage VLLMUploader completed in 397.02s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.32s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v2-q27b-lr1-74562-v8
Waiting for inference service chaiml-pony-v2-q27b-lr1-74562-v8 to be ready
2026-03-18T00:45:50.216136+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
2026-03-18T00:46:50.321242+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
2026-03-18T00:47:50.435014+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
Inference service chaiml-pony-v2-q27b-lr1-74562-v8 ready after 180.5295605659485s
Pipeline stage VLLMDeployer completed in 181.00s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-18T00:48:50.564205+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-18T00:49:50.672595+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.928889751434326s
Received healthy response to inference request in 4.60931921005249s
Received healthy response to inference request in 2.26310396194458s
Received healthy response to inference request in 4.584388971328735s
Received healthy response to inference request in 2.24147367477417s
2026-03-18T00:50:50.777889+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
Received healthy response to inference request in 2.3404922485351562s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.4319632053375244s
Received healthy response to inference request in 2.2600505352020264s
Received healthy response to inference request in 4.410460948944092s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.551910161972046s
Received healthy response to inference request in 2.2973647117614746s
2026-03-18T00:51:50.872692+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
Received healthy response to inference request in 17.927249670028687s
Received healthy response to inference request in 2.2871007919311523s
Received healthy response to inference request in 2.294922113418579s
Received healthy response to inference request in 2.3213815689086914s
Received healthy response to inference request in 2.621100425720215s
Received healthy response to inference request in 2.6229727268218994s
Received healthy response to inference request in 2.432199478149414s
Received healthy response to inference request in 2.4384593963623047s
Received healthy response to inference request in 2.394352436065674s
Received healthy response to inference request in 2.3410067558288574s
Received healthy response to inference request in 2.3807928562164307s
30 requests
8 failed requests
5th percentile: 2.2614245772361756
10th percentile: 2.284701108932495
20th percentile: 2.316578197479248
30th percentile: 2.3688570261001587
40th percentile: 2.432104969024658
50th percentile: 2.5865052938461304
60th percentile: 4.480032157897949
70th percentile: 8.828397727012597
80th percentile: 20.134643602371217
90th percentile: 20.15399796962738
95th percentile: 20.166968882083893
99th percentile: 20.186123850345613
mean time: 7.94055540561676
%s, retrying in %s seconds...
Received healthy response to inference request in 2.132431983947754s
Received healthy response to inference request in 2.1763696670532227s
Received healthy response to inference request in 2.248075008392334s
Received healthy response to inference request in 2.1319189071655273s
Received healthy response to inference request in 2.1860973834991455s
Received healthy response to inference request in 2.399163246154785s
Received healthy response to inference request in 2.1616499423980713s
Received healthy response to inference request in 2.2634899616241455s
Received healthy response to inference request in 2.2880585193634033s
2026-03-18T00:52:50.980170+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v8
Received healthy response to inference request in 2.2989954948425293s
Received healthy response to inference request in 2.230299711227417s
Received healthy response to inference request in 2.237632989883423s
Received healthy response to inference request in 2.2454988956451416s
Received healthy response to inference request in 2.262108325958252s
Received healthy response to inference request in 2.344511032104492s
Received healthy response to inference request in 2.284153938293457s
Received healthy response to inference request in 2.3290772438049316s
Received healthy response to inference request in 2.362264394760132s
Received healthy response to inference request in 2.6434075832366943s
Received healthy response to inference request in 2.379943609237671s
Received healthy response to inference request in 2.284701108932495s
Received healthy response to inference request in 2.5508170127868652s
Received healthy response to inference request in 2.211693048477173s
Received healthy response to inference request in 2.4288883209228516s
Received healthy response to inference request in 2.418765068054199s
Received healthy response to inference request in 2.3349924087524414s
Received healthy response to inference request in 2.3471286296844482s
Received healthy response to inference request in 2.3211755752563477s
Received healthy response to inference request in 2.3420650959014893s
Received healthy response to inference request in 2.437948703765869s
30 requests
0 failed requests
5th percentile: 2.145580065250397
10th percentile: 2.1748976945877074
20th percentile: 2.226578378677368
30th percentile: 2.2473021745681763
40th percentile: 2.2758883476257323
50th percentile: 2.2935270071029663
60th percentile: 2.3314433097839355
70th percentile: 2.345296311378479
80th percentile: 2.3837875366210937
90th percentile: 2.429794359207153
95th percentile: 2.500026273727417
99th percentile: 2.616556317806244
mean time: 2.3094440937042235
Pipeline stage StressChecker completed in 312.82s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.80s
Shutdown handler de-registered
chaiml-pony-v2-q27b-lr1_74562_v8 status is now deployed due to DeploymentManager action
chaiml-pony-v2-q27b-lr1_74562_v8 status is now inactive due to auto deactivation removed underperforming models