chaiml-pony-v3b-reverse_93453

developer_uid: zonemercy
submission_id: chaiml-pony-v3b-reverse_93453_v1
model_name: chaiml-pony-v3b-reverse_93453_v1
model_group: ChaiML/pony-v3b-reverse-
status: torndown
timestamp: 2026-04-02T10:21:15+00:00
num_battles: 11746
num_wins: 6374
celo_rating: 1327.33
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v3b-reverse-q27b-lr5e6ep2g8
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v3b-reverse_93453_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v3b-reverse-q27b-lr5e6ep2g8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-30
win_ratio: 0.5426528179805892
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '<|user|>', '####', '</s>', '<|assistant|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v3b-reverse-93453-v1-uploader
Waiting for job on chaiml-pony-v3b-reverse-93453-v1-uploader to finish
chaiml-pony-v3b-reverse-93453-v1-uploader: Using quantization_mode: fp8
chaiml-pony-v3b-reverse-93453-v1-uploader: Checking if ChaiML/pony-v3b-reverse-q27b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-pony-v3b-reverse-93453-v1-uploader: Downloading snapshot of ChaiML/pony-v3b-reverse-q27b-lr5e6ep2g8...
2026-03-30T07:34:49.492350+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
chaiml-pony-v3b-reverse-93453-v1-uploader: Downloaded in 51.022s
chaiml-pony-v3b-reverse-93453-v1-uploader: Loading /tmp/model_input...
chaiml-pony-v3b-reverse-93453-v1-uploader: The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
chaiml-pony-v3b-reverse-93453-v1-uploader: Applying quantization...
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:22.746645+0000 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:24.770022+0000 | reset | INFO - Compression lifecycle reset
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:24.773782+0000 | norm_calibration_context | INFO - Found 161 offset-norm modules to convert
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:24.782694+0000 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:24.827763+0000 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:24.828004+0000 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:24.840459+0000 | dispatch_model | WARNING - Forced to offload modules due to insufficient gpu resources
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:31.565216+0000 | norm_calibration_context | INFO - Restoring 161 norm modules to offset convention
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:31.571895+0000 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-pony-v3b-reverse-93453-v1-uploader: 2026-03-30T07:35:31.571985+0000 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-pony-v3b-reverse-93453-v1-uploader: Saving to /dev/shm/model_output...
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py:3344: UserWarning: Attempting to save a model with offloaded modules. Ensure that unallocated cpu memory exceeds the `shard_size` (50GB default)
chaiml-pony-v3b-reverse-93453-v1-uploader:   warnings.warn(
2026-03-30T07:35:49.594919+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
chaiml-pony-v3b-reverse-93453-v1-uploader: Updating config in /dev/shm/model_output
chaiml-pony-v3b-reverse-93453-v1-uploader: Pushing to ChaiML/pony-v3b-reverse-q27b-lr5e6ep2g8-FP8
chaiml-pony-v3b-reverse-93453-v1-uploader: Checking if ChaiML/pony-v3b-reverse-q27b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-pony-v3b-reverse-93453-v1-uploader: Creating repo ChaiML/pony-v3b-reverse-q27b-lr5e6ep2g8-FP8 and uploading /dev/shm/model_output to it
chaiml-pony-v3b-reverse-93453-v1-uploader: Found 1 files larger than 20GB (recommended limit):
chaiml-pony-v3b-reverse-93453-v1-uploader:   - model.safetensors: 35.9GB
chaiml-pony-v3b-reverse-93453-v1-uploader: Large files may slow down loading and processing.
chaiml-pony-v3b-reverse-93453-v1-uploader: ---------- 2026-03-30 07:36:21 (0:00:00) ----------
chaiml-pony-v3b-reverse-93453-v1-uploader: Files:   hashed 5/7 (34.3K/35.9G) | pre-uploaded: 0/0 (0.0/35.9G) (+7 unsure) | committed: 0/7 (0.0/35.9G) | ignored: 0
chaiml-pony-v3b-reverse-93453-v1-uploader: Workers: hashing: 2 | get upload mode: 5 | pre-uploading: 0 | committing: 0 | waiting: 57
chaiml-pony-v3b-reverse-93453-v1-uploader: ---------------------------------------------------
2026-03-30T07:36:49.691319+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
chaiml-pony-v3b-reverse-93453-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-pony-v3b-reverse-93453-v1-uploader: ---------- 2026-03-30 07:37:21 (0:01:00) ----------
chaiml-pony-v3b-reverse-93453-v1-uploader: Files:   hashed 7/7 (35.9G/35.9G) | pre-uploaded: 1/2 (20.0M/35.9G) | committed: 0/7 (0.0/35.9G) | ignored: 0
chaiml-pony-v3b-reverse-93453-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 63
chaiml-pony-v3b-reverse-93453-v1-uploader: ---------------------------------------------------
2026-03-30T07:37:50.206414+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
chaiml-pony-v3b-reverse-93453-v1-uploader:                              Processed model ChaiML/pony-v3b-reverse-q27b-lr5e6ep2g8 in 214.498s
chaiml-pony-v3b-reverse-93453-v1-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3b-reverse-93453-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v3b-reverse-93453-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3b-reverse-93453-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3b-reverse-93453-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3b-reverse-93453-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3b-reverse-93453-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v3b-reverse-93453-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v3b-reverse-93453-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v3b-reverse-93453-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v3b-reverse-93453-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v3b-reverse-93453-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v3b-reverse-93453-v1/default
chaiml-pony-v3b-reverse-93453-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v3b-reverse-93453-v1/default/chat_template.jinja
chaiml-pony-v3b-reverse-93453-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v3b-reverse-93453-v1/default/config.json
chaiml-pony-v3b-reverse-93453-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v3b-reverse-93453-v1/default/tokenizer_config.json
chaiml-pony-v3b-reverse-93453-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v3b-reverse-93453-v1/default/recipe.yaml
chaiml-pony-v3b-reverse-93453-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v3b-reverse-93453-v1/default/generation_config.json
chaiml-pony-v3b-reverse-93453-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v3b-reverse-93453-v1/default/tokenizer.json
2026-03-30T07:38:50.347116+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
chaiml-pony-v3b-reverse-93453-v1-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-pony-v3b-reverse-93453-v1/default/model.safetensors
Job chaiml-pony-v3b-reverse-93453-v1-uploader completed after 321.4s with status: succeeded
Stopping job with name chaiml-pony-v3b-reverse-93453-v1-uploader
Pipeline stage VLLMUploader completed in 321.88s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.90s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v3b-reverse-93453-v1
Waiting for inference service chaiml-pony-v3b-reverse-93453-v1 to be ready
2026-03-30T07:39:50.512891+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
2026-03-30T07:40:50.616881+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
2026-03-30T07:41:50.798723+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
Inference service chaiml-pony-v3b-reverse-93453-v1 ready after 162.48979544639587s
Pipeline stage VLLMDeployer completed in 163.03s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-30T07:42:50.902280+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-30T07:43:51.408942+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
Received healthy response to inference request in 15.45211386680603s
Received healthy response to inference request in 1.9022843837738037s
Received healthy response to inference request in 1.8070094585418701s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-30T07:44:51.566166+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.2351367473602295s
Received healthy response to inference request in 4.169504642486572s
Received healthy response to inference request in 2.706441640853882s
Received healthy response to inference request in 1.996088981628418s
Received healthy response to inference request in 1.9135384559631348s
Received healthy response to inference request in 2.2517313957214355s
Received healthy response to inference request in 1.9075541496276855s
Received healthy response to inference request in 1.9431068897247314s
Received healthy response to inference request in 1.9189887046813965s
Received healthy response to inference request in 4.166431188583374s
Received healthy response to inference request in 2.3405816555023193s
Received healthy response to inference request in 2.1298437118530273s
Received healthy response to inference request in 2.482651472091675s
Received healthy response to inference request in 2.3315470218658447s
Received healthy response to inference request in 2.0491139888763428s
Received healthy response to inference request in 1.974949836730957s
Received healthy response to inference request in 1.9620075225830078s
Received healthy response to inference request in 2.0253078937530518s
Received healthy response to inference request in 2.0044918060302734s
30 requests
8 failed requests
5th percentile: 1.9046557784080504
10th percentile: 1.9129400253295898
20th percentile: 1.9582273960113525
30th percentile: 2.001970958709717
40th percentile: 2.0975518226623535
50th percentile: 2.336064338684082
60th percentile: 3.2904374599456765
70th percentile: 7.600229883193938
80th percentile: 20.151754474639894
90th percentile: 20.186540174484254
95th percentile: 20.21486884355545
99th percentile: 20.453874773979187
mean time: 7.581721790631613
%s, retrying in %s seconds...
Received healthy response to inference request in 1.7474539279937744s
Received healthy response to inference request in 2.0281174182891846s
2026-03-30T07:45:51.701632+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
Received healthy response to inference request in 1.7884132862091064s
Received healthy response to inference request in 2.635286569595337s
Received healthy response to inference request in 1.827477216720581s
Received healthy response to inference request in 2.3538451194763184s
Received healthy response to inference request in 2.196044921875s
Received healthy response to inference request in 1.8539130687713623s
Received healthy response to inference request in 2.066481113433838s
Received healthy response to inference request in 2.010004997253418s
Received healthy response to inference request in 2.0254364013671875s
Received healthy response to inference request in 1.8809833526611328s
Received healthy response to inference request in 1.9784345626831055s
Received healthy response to inference request in 2.111651659011841s
Received healthy response to inference request in 1.9054582118988037s
Received healthy response to inference request in 1.8833537101745605s
Received healthy response to inference request in 2.0873117446899414s
Received healthy response to inference request in 1.9347078800201416s
Received healthy response to inference request in 1.9056200981140137s
Received healthy response to inference request in 1.9942810535430908s
Received healthy response to inference request in 1.9202771186828613s
Received healthy response to inference request in 2.019237756729126s
Received healthy response to inference request in 2.0672478675842285s
Received healthy response to inference request in 1.9479279518127441s
Received healthy response to inference request in 2.2279231548309326s
Received healthy response to inference request in 2.0305185317993164s
Received healthy response to inference request in 1.9776318073272705s
Received healthy response to inference request in 2.2188665866851807s
Received healthy response to inference request in 2.0369873046875s
Received healthy response to inference request in 2.7445435523986816s
30 requests
0 failed requests
5th percentile: 1.80599205493927
10th percentile: 1.8512694835662842
20th percentile: 1.9010373115539552
30th percentile: 1.9303786516189576
40th percentile: 1.9781134605407715
50th percentile: 2.014621376991272
2026-03-30T07:46:51.803123+00:00 monitor updated for chaiml-pony-v3b-reverse_93453_v1
60th percentile: 2.0290778636932374
70th percentile: 2.0667111396789553
80th percentile: 2.128530311584473
90th percentile: 2.240515351295471
95th percentile: 2.508637917041778
99th percentile: 2.712859027385712
mean time: 2.046847931543986
Pipeline stage StressChecker completed in 295.41s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.90s
Shutdown handler de-registered
chaiml-pony-v3b-reverse_93453_v1 status is now deployed due to DeploymentManager action
chaiml-pony-v3b-reverse_93453_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v3b-reverse_93453_v1 status is now torndown due to DeploymentManager action