developer_uid: zonemercy
submission_id: chaiml-pony-d3a-mv1-plc_89556_v3
model_name: chaiml-pony-d3a-mv1-plc_89556_v3
model_group: ChaiML/pony-d3a-mv1-plc-
status: inactive
timestamp: 2026-03-28T06:53:03+00:00
num_battles: 401
num_wins: 179
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep2g8
model_architecture: Qwen3_5MoeForConditionalGeneration
model_num_parameters: 33753909248.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3a-mv1-plc_89556_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep2g8
model_size: 34B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.4463840399002494
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.8, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '####', '</s>', '<|assistant|>', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3a-mv1-plc-89556-v3-uploader
Waiting for job on chaiml-pony-d3a-mv1-plc-89556-v3-uploader to finish
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: Using quantization_mode: fp8
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: Checking if ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: Downloading snapshot of ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep2g8-FP8...
2026-03-28T06:30:04.055274+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: Downloaded in 37.819s
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: Processed model ChaiML/pony-d3a-mv1-plc-q35b-lr5e6ep2g8 in 40.308s
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-89556-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3a-mv1-plc-89556-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-89556-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-89556-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-89556-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3a-mv1-plc-89556-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3a-mv1-plc-89556-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3a-mv1-plc-89556-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default/.gitattributes
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default/generation_config.json
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default/chat_template.jinja
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default/tokenizer_config.json
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default/config.json
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default/recipe.yaml
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default/tokenizer.json
2026-03-28T06:31:04.144951+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
chaiml-pony-d3a-mv1-plc-89556-v3-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-d3a-mv1-plc-89556-v3/default/model.safetensors
Job chaiml-pony-d3a-mv1-plc-89556-v3-uploader completed after 163.56s with status: succeeded
Stopping job with name chaiml-pony-d3a-mv1-plc-89556-v3-uploader
Pipeline stage VLLMUploader completed in 164.26s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.86s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3a-mv1-plc-89556-v3
Waiting for inference service chaiml-pony-d3a-mv1-plc-89556-v3 to be ready
2026-03-28T06:32:04.241592+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
2026-03-28T06:33:04.339233+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
2026-03-28T06:34:04.489528+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
2026-03-28T06:35:04.584757+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
Inference service chaiml-pony-d3a-mv1-plc-89556-v3 ready after 201.292622089386s
Pipeline stage VLLMDeployer completed in 201.70s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T06:36:04.677289+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 10.637468576431274s
Received healthy response to inference request in 1.6611011028289795s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T06:37:04.778953+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-d3a-mv1-plc-89556-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination')"}
Received unhealthy response to inference request!
Received healthy response to inference request in 4.814144611358643s
Received healthy response to inference request in 1.1976091861724854s
Received healthy response to inference request in 3.623326301574707s
Received healthy response to inference request in 1.1727910041809082s
Received healthy response to inference request in 1.066572666168213s
Received healthy response to inference request in 1.232302188873291s
Received healthy response to inference request in 1.112924575805664s
Received healthy response to inference request in 1.1053955554962158s
2026-03-28T06:38:04.868978+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.1933836936950684s
Received healthy response to inference request in 1.1936328411102295s
Received healthy response to inference request in 1.3119251728057861s
Received healthy response to inference request in 1.0821917057037354s
Failed to get response for submission chaiml-qwen-bobo-dpo-ju_56781_v4: ('http://chaiml-qwen-bobo-dpo-ju-56781-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'request timeout')
Received healthy response to inference request in 1.113049030303955s
Received healthy response to inference request in 1.087890386581421s
Received healthy response to inference request in 1.207204818725586s
Received healthy response to inference request in 1.1159107685089111s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.0903456211090088s
Received healthy response to inference request in 1.2059526443481445s
Received healthy response to inference request in 1.1305480003356934s
30 requests
9 failed requests
5th percentile: 1.084756112098694
10th percentile: 1.09010009765625
20th percentile: 1.1130241394042968
30th percentile: 1.1601181030273438
40th percentile: 1.196018648147583
50th percentile: 1.2197535037994385
60th percentile: 2.4459911823272678
70th percentile: 7.399037981033312
80th percentile: 20.12288646697998
90th percentile: 20.133703446388246
95th percentile: 20.14529538154602
99th percentile: 20.29542311191559
mean time: 6.921431342760722
%s, retrying in %s seconds...
Received healthy response to inference request in 16.488296270370483s
Received healthy response to inference request in 1.196448564529419s
Received healthy response to inference request in 1.0234479904174805s
Received healthy response to inference request in 1.4629502296447754s
2026-03-28T06:39:04.968512+00:00 monitor updated for chaiml-pony-d3a-mv1-plc_89556_v3
Received healthy response to inference request in 1.580162525177002s
Received healthy response to inference request in 1.2092564105987549s
Received healthy response to inference request in 1.4237236976623535s
Received healthy response to inference request in 1.3725461959838867s
Received healthy response to inference request in 1.073620319366455s
Received healthy response to inference request in 1.2173030376434326s
Received healthy response to inference request in 1.5043072700500488s
Received healthy response to inference request in 1.4900193214416504s
Received healthy response to inference request in 1.6724579334259033s
Received healthy response to inference request in 1.0769760608673096s
Received healthy response to inference request in 1.176753282546997s
Received healthy response to inference request in 1.2604987621307373s
Received healthy response to inference request in 1.1061842441558838s
Received healthy response to inference request in 1.218656063079834s
Received healthy response to inference request in 1.0893666744232178s
Received healthy response to inference request in 1.1706135272979736s
Received healthy response to inference request in 1.1397840976715088s
Failed to get request counts for guanaco-submitter. Falling back to default
Received healthy response to inference request in 1.0909662246704102s
Received healthy response to inference request in 1.1454927921295166s
Received healthy response to inference request in 1.1366822719573975s
Received healthy response to inference request in 1.1103663444519043s
Received healthy response to inference request in 1.2458224296569824s
Received healthy response to inference request in 1.3539292812347412s
Received healthy response to inference request in 1.5781869888305664s
Received healthy response to inference request in 1.1887602806091309s
Received healthy response to inference request in 1.1378910541534424s
30 requests
0 failed requests
5th percentile: 1.0751304030418396
10th percentile: 1.088127613067627
20th percentile: 1.1095299243927002
30th percentile: 1.1392161846160889
40th percentile: 1.1742973804473877
50th percentile: 1.202852487564087
60th percentile: 1.2295226097106933
70th percentile: 1.3595143556594849
80th percentile: 1.4683640480041504
90th percentile: 1.5783845424652099
95th percentile: 1.6309249997138975
99th percentile: 12.191703152656569
mean time: 1.7647156715393066
Pipeline stage StressChecker completed in 266.95s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.80s
Shutdown handler de-registered
chaiml-pony-d3a-mv1-plc_89556_v3 status is now deployed due to DeploymentManager action
chaiml-pony-d3a-mv1-plc_89556_v3 status is now inactive due to admin request