developer_uid: zonemercy
submission_id: chaiml-pony-d3-g46-o4-l_75766_v1
model_name: chaiml-pony-d3-g46-o4-l_75766_v1
model_group: ChaiML/pony-d3-g46-o4-lr
status: torndown
timestamp: 2026-03-14T07:02:03+00:00
num_battles: 11290
num_wins: 6355
celo_rating: 1342.2
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3-g46-o4-l_75766_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-10
win_ratio: 0.562887511071745
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3-g46-o4-l-75766-v1-uploader
Waiting for job on chaiml-pony-d3-g46-o4-l-75766-v1-uploader to finish
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Using quantization_mode: fp8
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Checking if ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8-FP8 already exists in ChaiML
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Downloading snapshot of ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8...
2026-03-11T04:30:41.539340+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:31:41.628111+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:32:41.719277+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:33:41.804758+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:34:41.887460+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Downloaded in 253.012s
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Loading /tmp/model_input...
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Applying quantization...
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: 2026-03-10T21:35:16.174234-0700 | reset | INFO - Compression lifecycle reset
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: 2026-03-10T21:35:16.248469-0700 | from_modifiers | INFO - Creating recipe from modifiers
2026-03-11T04:35:41.976599+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: 2026-03-10T21:36:13.710996-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: 2026-03-10T21:36:13.711484-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
2026-03-11T04:36:42.072041+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
2026-03-11T04:37:42.163063+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:38:42.251392+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:39:42.342288+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:40:42.428270+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:41:42.516182+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: 2026-03-10T21:42:00.002148-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: 2026-03-10T21:42:07.602828-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Saving to /dev/shm/model_output...
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: 2026-03-10T21:42:07.652008-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
2026-03-11T04:42:42.603205+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:43:42.697769+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:44:42.822789+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:45:42.937435+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:46:43.028029+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:47:43.139056+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:48:43.227197+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:49:43.330388+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:50:43.444139+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:51:43.557415+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:52:43.651168+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:53:43.745025+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:54:43.851952+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:55:43.946047+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:56:44.040667+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:57:44.155208+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:58:44.242166+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T04:59:44.330302+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:00:44.461045+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:01:44.553289+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:02:44.643057+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:03:44.739132+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:04:44.829377+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:05:44.919766+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:06:45.017902+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:07:45.112955+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:08:45.222120+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:09:45.315938+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:10:45.426842+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:11:45.521418+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:12:45.612351+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:13:45.701261+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:14:45.794254+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:15:45.909793+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:16:46.024256+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:17:46.117014+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:18:46.205447+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:19:46.300552+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:20:46.391175+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:21:46.485543+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:22:46.587008+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:23:46.678644+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Pushing to ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8-FP8
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Checking if ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8-FP8 already exists in ChaiML
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Creating repo ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8-FP8 and uploading /dev/shm/model_output to it
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:24:01 (0:00:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:24:46.790281+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: 







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:25:01 (0:01:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:25:46.884661+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:26:01 (0:02:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:26:46.977012+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:27:01 (0:03:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 14/73 (60.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 59 | committing: 0 | waiting: 67
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:27:47.135086+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:28:01 (0:04:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 19/73 (85.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 54 | committing: 0 | waiting: 72
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:28:47.234185+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:29:01 (0:05:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (124.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:29:47.409828+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:30:01 (0:06:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:30:47.535704+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:31:01 (0:07:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:31:47.635207+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:32:01 (0:08:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 50/73 (239.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 23 | committing: 0 | waiting: 103
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:32:47.757528+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:33:01 (0:09:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 51/73 (244.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 22 | committing: 0 | waiting: 104
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:33:47.859440+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:34:01 (0:10:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 59/73 (284.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 14 | committing: 0 | waiting: 112
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:34:47.959140+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:35:01 (0:11:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 67/73 (324.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 6 | committing: 0 | waiting: 120
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:35:48.080978+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:36:01 (0:12:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
2026-03-11T05:36:48.210923+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              







chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------- 2026-03-10 22:37:02 (0:13:00) ----------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.8G/354.7G) | ignored: 0
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:                              Processed model ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8 in 4038.399s
2026-03-11T05:37:48.313706+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3-g46-o4-l-75766-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/recipe.yaml
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/generation_config.json
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/chat_template.jinja
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/tokenizer_config.json
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/special_tokens_map.json
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/tokenizer.json
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/config.json
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model.safetensors.index.json
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00072-of-00072.safetensors
2026-03-11T05:38:48.431114+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00071-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00006-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00007-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00001-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00031-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00020-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00038-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00059-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00025-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00035-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00002-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00057-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00056-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00010-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00004-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00003-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00026-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00043-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00029-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00016-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00068-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00011-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00055-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00062-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00009-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00058-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00051-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00054-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00065-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00027-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00042-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00033-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00013-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00044-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00030-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00039-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00017-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00018-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00045-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00040-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00060-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00063-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00037-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00048-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00015-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00061-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00032-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00034-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00008-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00036-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00041-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00067-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00028-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00022-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00005-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00047-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00050-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00069-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00012-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v1/default/model-00064-of-00072.safetensors
Job chaiml-pony-d3-g46-o4-l-75766-v1-uploader completed after 4200.92s with status: succeeded
Stopping job with name chaiml-pony-d3-g46-o4-l-75766-v1-uploader
Pipeline stage VLLMUploader completed in 4201.46s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.82s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3-g46-o4-l-75766-v1
Waiting for inference service chaiml-pony-d3-g46-o4-l-75766-v1 to be ready
2026-03-11T05:39:48.540281+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:40:48.665306+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:41:48.826881+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:42:48.982773+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:43:49.152159+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:44:49.264392+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:45:49.420164+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
2026-03-11T05:46:49.711977+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
Inference service chaiml-pony-d3-g46-o4-l-75766-v1 ready after 430.63866686820984s
Pipeline stage VLLMDeployer completed in 431.14s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.102951288223267s
Received healthy response to inference request in 8.998523235321045s
Received healthy response to inference request in 1.958918571472168s
Received healthy response to inference request in 1.9463591575622559s
Received healthy response to inference request in 1.967172384262085s
Received healthy response to inference request in 9.236085653305054s
Received healthy response to inference request in 2.0331499576568604s
Received healthy response to inference request in 2.0351579189300537s
Received healthy response to inference request in 9.225050926208496s
Received healthy response to inference request in 1.9163427352905273s
Received healthy response to inference request in 1.9605014324188232s
Received healthy response to inference request in 1.9482896327972412s
2026-03-11T05:47:49.827930+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v1
Received healthy response to inference request in 2.017465591430664s
Received healthy response to inference request in 2.058276653289795s
Received healthy response to inference request in 2.0057950019836426s
Received healthy response to inference request in 1.954803466796875s
Received healthy response to inference request in 1.9628345966339111s
Received healthy response to inference request in 2.0335702896118164s
Received healthy response to inference request in 2.0086400508880615s
Received healthy response to inference request in 1.956899881362915s
Received healthy response to inference request in 1.9710643291473389s
Received healthy response to inference request in 2.0105233192443848s
Received healthy response to inference request in 1.9751341342926025s
Received healthy response to inference request in 1.942481279373169s
Received healthy response to inference request in 1.9707872867584229s
Received healthy response to inference request in 2.0324220657348633s
Received healthy response to inference request in 1.9842860698699951s
Received healthy response to inference request in 2.0392019748687744s
Received healthy response to inference request in 1.987581491470337s
Received healthy response to inference request in 2.0025410652160645s
30 requests
0 failed requests
5th percentile: 1.944226324558258
10th percentile: 1.9480965852737426
20th percentile: 1.9585148334503173
30th percentile: 1.9658710479736328
40th percentile: 1.973506212234497
50th percentile: 1.9950612783432007
60th percentile: 2.0093933582305907
70th percentile: 2.0326404333114625
80th percentile: 2.035966730117798
90th percentile: 9.008966040611266
95th percentile: 9.170106089115142
99th percentile: 9.232885582447052
mean time: 2.9414270480473834
Pipeline stage StressChecker completed in 91.86s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-pony-d3-g46-o4-l_75766_v1 status is now deployed due to DeploymentManager action
chaiml-pony-d3-g46-o4-l_75766_v1 status is now inactive due to auto deactivation removed underperforming models
Deleting key chaiml-pony-d3-g46-o4-l-49714-v1/default/tokenizer.json from bucket guanaco-vllm-models
chaiml-pony-d3-g46-o4-l_75766_v1 status is now torndown due to DeploymentManager action