developer_uid: zonemercy
submission_id: chaiml-pony-d3-g46-o4-l_75766_v3
model_name: chaiml-pony-d3-g46-o4-l_75766_v3
model_group: ChaiML/pony-d3-g46-o4-lr
status: torndown
timestamp: 2026-03-11T09:47:44+00:00
num_battles: 10432
num_wins: 5943
celo_rating: 1343.32
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3-g46-o4-l_75766_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-11
win_ratio: 0.5696894171779141
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3-g46-o4-l-75766-v3-uploader
Waiting for job on chaiml-pony-d3-g46-o4-l-75766-v3-uploader to finish
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: Using quantization_mode: fp8
2026-03-11T07:47:07.130463+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: Checking if ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8-FP8 already exists in ChaiML
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: Downloading snapshot of ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8-FP8...
2026-03-11T07:48:07.213273+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: Downloaded in 108.432s
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: Processed model ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8 in 111.968s
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3-g46-o4-l-75766-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3-g46-o4-l-75766-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/config.json
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/generation_config.json
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/recipe.yaml
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/.gitattributes
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/tokenizer_config.json
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/special_tokens_map.json
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model.safetensors.index.json
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/tokenizer.json
2026-03-11T07:49:07.293058+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00072-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: 	status code: 500, request id: 2b17878e-9359-98c0-a000-305454f54556, host id: 
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: 	status code: 500, request id: 4c493a56-7d8b-9bef-89c9-70e0554a3360, host id: 
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00071-of-00072.safetensors
2026-03-11T07:50:07.379655+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00043-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00047-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00062-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00010-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00066-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00069-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00050-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00030-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00061-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00015-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00042-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00057-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00026-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00053-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00070-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00003-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00018-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00017-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00034-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00063-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00025-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00002-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00049-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00032-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00013-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00014-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00052-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00008-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00027-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00059-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00007-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00029-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00040-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00005-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00036-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00051-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00048-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00044-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00009-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00058-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00064-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00001-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00004-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00016-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00019-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00031-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00038-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00060-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00041-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00022-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00020-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00067-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00046-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00039-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00006-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00068-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00012-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00024-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00035-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00054-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00037-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00028-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00056-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00033-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00055-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v3-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v3/default/model-00021-of-00072.safetensors
Job chaiml-pony-d3-g46-o4-l-75766-v3-uploader completed after 267.25s with status: succeeded
Stopping job with name chaiml-pony-d3-g46-o4-l-75766-v3-uploader
Pipeline stage VLLMUploader completed in 267.70s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.34s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3-g46-o4-l-75766-v3
Waiting for inference service chaiml-pony-d3-g46-o4-l-75766-v3 to be ready
2026-03-11T07:51:07.470313+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
2026-03-11T07:52:07.568567+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
2026-03-11T07:53:07.659558+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
2026-03-11T07:54:07.754426+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
2026-03-11T07:55:07.847789+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
2026-03-11T07:56:07.945609+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
2026-03-11T07:57:08.039052+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
Inference service chaiml-pony-d3-g46-o4-l-75766-v3 ready after 441.042875289917s
Pipeline stage VLLMDeployer completed in 441.51s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.300023794174194s
2026-03-11T07:58:08.139441+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
Received healthy response to inference request in 1.9768311977386475s
Received healthy response to inference request in 1.9303112030029297s
Received healthy response to inference request in 2.0160956382751465s
Received healthy response to inference request in 9.139929056167603s
Received healthy response to inference request in 2.034947633743286s
Received healthy response to inference request in 8.97799015045166s
Received healthy response to inference request in 2.367788791656494s
Received healthy response to inference request in 2.0309178829193115s
Received healthy response to inference request in 1.998396873474121s
Received healthy response to inference request in 1.9750878810882568s
Received healthy response to inference request in 9.509551525115967s
Received healthy response to inference request in 1.9598991870880127s
Received healthy response to inference request in 1.9854068756103516s
Received healthy response to inference request in 1.9610583782196045s
Received healthy response to inference request in 2.0310988426208496s
Received healthy response to inference request in 2.0188677310943604s
Received healthy response to inference request in 2.061516761779785s
Received healthy response to inference request in 1.9815170764923096s
Received healthy response to inference request in 1.958348035812378s
2026-03-11T07:59:08.246666+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v3
Received healthy response to inference request in 1.9687135219573975s
Received healthy response to inference request in 2.0501248836517334s
Received healthy response to inference request in 1.9723072052001953s
Received healthy response to inference request in 2.0021185874938965s
Received healthy response to inference request in 1.9749877452850342s
Received healthy response to inference request in 2.295527219772339s
Received healthy response to inference request in 2.0934646129608154s
Received healthy response to inference request in 1.9642746448516846s
Received healthy response to inference request in 1.9797816276550293s
Received healthy response to inference request in 1.9940872192382812s
30 requests
0 failed requests
5th percentile: 1.9590460538864136
10th percentile: 1.9609424591064453
20th percentile: 1.9715884685516358
30th percentile: 1.9763082027435304
40th percentile: 1.9838509559631348
50th percentile: 2.000257730484009
60th percentile: 2.023687791824341
70th percentile: 2.0395008087158204
80th percentile: 2.1338771343231207
90th percentile: 8.994184041023255
95th percentile: 9.227981162071227
99th percentile: 9.448788483142852
mean time: 2.9836990594863892
Pipeline stage StressChecker completed in 92.47s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.72s
Shutdown handler de-registered
chaiml-pony-d3-g46-o4-l_75766_v3 status is now deployed due to DeploymentManager action
chaiml-pony-d3-g46-o4-l_75766_v3 status is now inactive due to auto deactivation removed underperforming models