chaiml-pony-d3-g46-pv2_41910

developer_uid: chai_backend_admin
submission_id: chaiml-pony-d3-g46-pv2_41910_v14
model_name: chaiml-pony-d3-g46-pv2_41910_v14
model_group: ChaiML/pony-d3-g46-pv2-l
status: inactive
timestamp: 2026-03-27T09:16:49+00:00
num_battles: 12234
num_wins: 7029
celo_rating: 1346.25
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3-g46-pv2_41910_v14
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5745463462481608
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '</s>', '<|user|>', '####', '<|assistant|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3-g46-pv2-41910-v14-uploader
Waiting for job on chaiml-pony-d3-g46-pv2-41910-v14-uploader to finish
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Using quantization_mode: w4a16
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Checking if ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8-W4A16 already exists in ChaiML
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Downloading snapshot of ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8...
2026-03-27T06:53:47.891345+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T06:54:48.092193+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T06:55:48.302642+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T06:56:48.534925+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Downloaded in 231.434s
2026-03-27T06:57:48.726956+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T06:58:48.922075+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T06:59:49.109460+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T07:00:49.308656+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T07:01:49.515982+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Applying quantization...
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:02:37 INFO __init__.py L202: Patched transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeMoE -> auto_round.modeling.unfused_moe.glm_moe.LinearGlm4MoeMoE[0m
2026-03-27T07:02:49.714714+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:03:17 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:03:34 INFO device.py L1468:  'peak_ram': 16.6GB, 'peak_vram': 1.44GB[0m
2026-03-27T07:03:49.924593+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:03:46 INFO device.py L1468:  'peak_ram': 21.05GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:04:02 INFO device.py L1468:  'peak_ram': 26.28GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:04:13 INFO device.py L1468:  'peak_ram': 26.28GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:04:29 INFO device.py L1468:  'peak_ram': 26.28GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:04:40 INFO device.py L1468:  'peak_ram': 26.88GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:04:50.111276+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:04:51 INFO device.py L1468:  'peak_ram': 26.88GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:05:05 INFO device.py L1468:  'peak_ram': 27.43GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:05:16 INFO device.py L1468:  'peak_ram': 27.43GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:05:32 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:05:40 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:05:46 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:05:50.314458+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:05:54 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:00 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:05 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:14 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:19 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:28 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:33 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:39 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:06:50.514232+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:48 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:06:54 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:07:09 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:07:16 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:07:25 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:07:31 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:07:37 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:07:50.708425+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:07:46 INFO device.py L1468:  'peak_ram': 28.06GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:07:51 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:08:00 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:08:06 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:08:11 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:08:19 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:08:24 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:08:30 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:08:38 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:08:50.975498+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:08:49 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:09:04 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:09:15 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:09:27 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:09:44 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:09:51.192376+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:09:57 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:10:17 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:10:32 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:10:43 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:10:51.418649+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:10:57 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:11:08 INFO device.py L1468:  'peak_ram': 34.65GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:11:19 INFO device.py L1468:  'peak_ram': 35.23GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:11:33 INFO device.py L1468:  'peak_ram': 35.23GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:11:51.896256+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:11:57 INFO device.py L1468:  'peak_ram': 35.23GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:12:11 INFO device.py L1468:  'peak_ram': 35.23GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:12:24 INFO device.py L1468:  'peak_ram': 35.23GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:12:41 INFO device.py L1468:  'peak_ram': 35.23GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:12:52.098707+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:12:52 INFO device.py L1468:  'peak_ram': 35.3GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:13:06 INFO device.py L1468:  'peak_ram': 35.3GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:13:17 INFO device.py L1468:  'peak_ram': 35.3GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:13:28 INFO device.py L1468:  'peak_ram': 35.3GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:13:41 INFO device.py L1468:  'peak_ram': 41.73GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:13:52.293497+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:13:52 INFO device.py L1468:  'peak_ram': 41.73GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:14:04 INFO device.py L1468:  'peak_ram': 42.62GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:14:18 INFO device.py L1468:  'peak_ram': 42.62GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:14:29 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:14:43 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:14:52.494859+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:14:55 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:15:05 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:15:20 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:15:31 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:15:52.859718+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:15:45 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:15:58 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:16:13 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:16:29 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:16:43 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:16:53.057303+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:16:58 INFO device.py L1468:  'peak_ram': 43.01GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:17:15 INFO device.py L1468:  'peak_ram': 44.2GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:17:45 INFO device.py L1468:  'peak_ram': 44.2GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:17:53.251607+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:18:00 INFO device.py L1468:  'peak_ram': 44.2GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:18:14 INFO device.py L1468:  'peak_ram': 44.2GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:18:31 INFO device.py L1468:  'peak_ram': 44.2GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:18:43 INFO device.py L1468:  'peak_ram': 44.2GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:18:53.451897+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:18:58 INFO device.py L1468:  'peak_ram': 45.24GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:19:16 INFO device.py L1468:  'peak_ram': 45.24GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:19:31 INFO device.py L1468:  'peak_ram': 45.24GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:19:53.654911+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:19:47 INFO device.py L1468:  'peak_ram': 45.24GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:20:01 INFO device.py L1468:  'peak_ram': 45.24GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:20:16 INFO device.py L1468:  'peak_ram': 45.24GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:20:32 INFO device.py L1468:  'peak_ram': 45.24GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:20:45 INFO device.py L1468:  'peak_ram': 49.8GB, 'peak_vram': 1.59GB[0m
2026-03-27T07:20:53.850783+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:20:57 INFO device.py L1468:  'peak_ram': 49.8GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:21:07 INFO device.py L1468:  'peak_ram': 51.11GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:21:16 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [33;1m2026-03-27 07:21:16 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: [38;20m2026-03-27 07:21:16 INFO device.py L1468:  'peak_ram': 51.11GB, 'peak_vram': 1.59GB[0m
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Checking if ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8-W4A16 already exists in ChaiML
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Creating repo ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8-W4A16 and uploading /dev/shm/model_output to it
chaiml-pony-d3-g46-pv2-41910-v14-uploader: ---------- 2026-03-27 07:21:17 (0:00:00) ----------
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/1 (0.0/197.4G) (+43 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Workers: hashing: 38 | get upload mode: 2 | pre-uploading: 0 | committing: 0 | waiting: 24
chaiml-pony-d3-g46-pv2-41910-v14-uploader: ---------------------------------------------------
2026-03-27T07:21:54.059169+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-pony-d3-g46-pv2-41910-v14-uploader: ---------- 2026-03-27 07:22:17 (0:01:00) ----------
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 13/40 (52.5G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 27 | committing: 0 | waiting: 37
chaiml-pony-d3-g46-pv2-41910-v14-uploader: ---------------------------------------------------
2026-03-27T07:22:54.265137+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-pony-d3-g46-pv2-41910-v14-uploader: ---------- 2026-03-27 07:23:17 (0:02:00) ----------
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-pony-d3-g46-pv2-41910-v14-uploader: ---------------------------------------------------
2026-03-27T07:23:54.484107+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
chaiml-pony-d3-g46-pv2-41910-v14-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3-g46-pv2-41910-v14-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-41910-v14-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3-g46-pv2-41910-v14-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3-g46-pv2-41910-v14-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3-g46-pv2-41910-v14-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-41910-v14-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-pv2-41910-v14-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-41910-v14-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-pv2-41910-v14-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-41910-v14-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-pv2-41910-v14-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-41910-v14-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-pv2-41910-v14-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3-g46-pv2-41910-v14-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3-g46-pv2-41910-v14-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3-g46-pv2-41910-v14-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3-g46-pv2-41910-v14-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3-g46-pv2-41910-v14-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/chat_template.jinja
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/quantization_config.json
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/tokenizer_config.json
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/config.json
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model.safetensors.index.json
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/generation_config.json
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/tokenizer.json
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00038-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00036-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00037-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00021-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00007-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00010-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00013-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00012-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00019-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00024-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00011-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00020-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00023-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00008-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00029-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00018-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00032-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00001-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00006-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00014-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00009-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00035-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00015-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00026-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00030-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00031-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00034-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00002-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00005-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00004-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00016-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00028-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00033-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00003-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00025-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00017-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00022-of-00038.safetensors
chaiml-pony-d3-g46-pv2-41910-v14-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-41910-v14/default/model-00027-of-00038.safetensors
Job chaiml-pony-d3-g46-pv2-41910-v14-uploader completed after 1918.18s with status: succeeded
Stopping job with name chaiml-pony-d3-g46-pv2-41910-v14-uploader
Pipeline stage VLLMUploader completed in 1919.19s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.23s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.30s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3-g46-pv2-41910-v14
Waiting for inference service chaiml-pony-d3-g46-pv2-41910-v14 to be ready
2026-03-27T07:24:54.806112+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T07:25:55.018411+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T07:26:55.256551+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T07:27:55.479260+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
2026-03-27T07:28:55.705470+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
Inference service chaiml-pony-d3-g46-pv2-41910-v14 ready after 283.09054803848267s
Pipeline stage VLLMDeployer completed in 284.35s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.741864442825317s
Received healthy response to inference request in 10.199958324432373s
2026-03-27T07:29:55.925494+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
Received healthy response to inference request in 2.2543866634368896s
Received healthy response to inference request in 9.27164602279663s
Received healthy response to inference request in 2.212974786758423s
Received healthy response to inference request in 8.88939118385315s
Received healthy response to inference request in 9.29451847076416s
Received healthy response to inference request in 2.1221208572387695s
Received healthy response to inference request in 2.139725923538208s
Received healthy response to inference request in 2.335988759994507s
Received healthy response to inference request in 2.528947353363037s
Received healthy response to inference request in 2.18090558052063s
Received healthy response to inference request in 2.104329824447632s
Received healthy response to inference request in 2.3729376792907715s
Received healthy response to inference request in 2.568047046661377s
Received healthy response to inference request in 2.224796772003174s
Received healthy response to inference request in 2.604236602783203s
Received healthy response to inference request in 2.1584744453430176s
Received healthy response to inference request in 2.3345518112182617s
2026-03-27T07:30:56.148680+00:00 monitor updated for chaiml-pony-d3-g46-pv2_41910_v14
Received healthy response to inference request in 2.1027073860168457s
Received healthy response to inference request in 2.141101121902466s
Received healthy response to inference request in 2.212388515472412s
Received healthy response to inference request in 2.107924699783325s
Received healthy response to inference request in 2.1670730113983154s
Received healthy response to inference request in 2.22908353805542s
Received healthy response to inference request in 2.2246832847595215s
Received healthy response to inference request in 2.239175796508789s
Received healthy response to inference request in 2.2343106269836426s
Received healthy response to inference request in 2.2438132762908936s
Received healthy response to inference request in 2.33738374710083s
30 requests
0 failed requests
5th percentile: 2.1059475183486938
10th percentile: 2.120701241493225
20th percentile: 2.154999780654907
30th percentile: 2.2029436349868776
40th percentile: 2.224751377105713
50th percentile: 2.236743211746216
60th percentile: 2.286452722549438
70th percentile: 2.3480499267578123
80th percentile: 2.575284957885742
90th percentile: 8.927616667747499
95th percentile: 9.284225869178773
99th percentile: 9.937380766868593
mean time: 3.4259815851847333
Pipeline stage StressChecker completed in 109.02s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.48s
Shutdown handler de-registered
chaiml-pony-d3-g46-pv2_41910_v14 status is now deployed due to DeploymentManager action
chaiml-pony-d3-g46-pv2_41910_v14 status is now inactive due to auto deactivation removed underperforming models