developer_uid: zonemercy
submission_id: chaiml-pony-d3-g46-o4-l_75766_v2
model_name: chaiml-pony-d3-g46-o4-l_75766_v2
model_group: ChaiML/pony-d3-g46-o4-lr
status: torndown
timestamp: 2026-03-14T10:01:58+00:00
num_battles: 11776
num_wins: 6643
celo_rating: 1339.6
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3-g46-o4-l_75766_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-11
win_ratio: 0.5641134510869565
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3-g46-o4-l-75766-v2-uploader
Waiting for job on chaiml-pony-d3-g46-o4-l-75766-v2-uploader to finish
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: Using quantization_mode: fp8
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: Checking if ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8-FP8 already exists in ChaiML
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: Downloading snapshot of ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8-FP8...
2026-03-11T07:46:56.324758+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
2026-03-11T07:47:56.407558+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: Downloaded in 128.457s
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: Processed model ChaiML/pony-d3-g46-o4-lr5e6ep2r64b8 in 131.905s
2026-03-11T07:48:56.509201+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3-g46-o4-l-75766-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-o4-l-75766-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3-g46-o4-l-75766-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3-g46-o4-l-75766-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/generation_config.json
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/config.json
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/chat_template.jinja
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/.gitattributes
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/recipe.yaml
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/special_tokens_map.json
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/tokenizer_config.json
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model.safetensors.index.json
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/tokenizer.json
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00072-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: 	status code: 500, request id: 436f5010-feae-94bb-9007-f1f72c34c9d8, host id: 
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: 	status code: 500, request id: 6b4010e1-f6a2-95c5-a984-1f06c2f8c968, host id: 
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00071-of-00072.safetensors
2026-03-11T07:49:56.680388+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00064-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00046-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00070-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00042-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00017-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00059-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00057-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00029-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00069-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00025-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00051-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00061-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00016-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00027-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00018-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00033-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00032-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00024-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00060-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00001-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00011-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00041-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00004-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00015-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00020-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00063-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00052-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00039-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00009-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00050-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00066-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00055-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00044-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00049-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00037-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00014-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00062-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00043-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00012-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00036-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00010-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00022-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00005-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00034-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00035-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00007-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00013-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00023-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00058-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00065-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00026-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00054-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00045-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00040-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00008-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00038-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00030-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00067-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00019-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00031-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00002-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00003-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00021-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00053-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00056-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00028-of-00072.safetensors
chaiml-pony-d3-g46-o4-l-75766-v2-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-o4-l-75766-v2/default/model-00047-of-00072.safetensors
Job chaiml-pony-d3-g46-o4-l-75766-v2-uploader completed after 277.15s with status: succeeded
Stopping job with name chaiml-pony-d3-g46-o4-l-75766-v2-uploader
Pipeline stage VLLMUploader completed in 277.61s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.70s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3-g46-o4-l-75766-v2
Waiting for inference service chaiml-pony-d3-g46-o4-l-75766-v2 to be ready
2026-03-11T07:50:56.803780+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
2026-03-11T07:51:56.903270+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
2026-03-11T07:52:56.993438+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
2026-03-11T07:53:57.085354+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
2026-03-11T07:54:57.175503+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
2026-03-11T07:55:57.268198+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
Inference service chaiml-pony-d3-g46-o4-l-75766-v2 ready after 371.51155161857605s
Pipeline stage VLLMDeployer completed in 371.92s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.11888074874878s
2026-03-11T07:56:57.359552+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
Received healthy response to inference request in 1.9896812438964844s
Received healthy response to inference request in 2.0620229244232178s
Received healthy response to inference request in 2.0293192863464355s
Received healthy response to inference request in 1.9198684692382812s
Received healthy response to inference request in 2.0456981658935547s
Received healthy response to inference request in 8.98086953163147s
Received healthy response to inference request in 1.9615652561187744s
Received healthy response to inference request in 1.971384048461914s
Received healthy response to inference request in 1.964820384979248s
Received healthy response to inference request in 1.978072166442871s
Received healthy response to inference request in 2.317342758178711s
Received healthy response to inference request in 2.038048267364502s
Received healthy response to inference request in 2.0439393520355225s
Received healthy response to inference request in 1.9473419189453125s
Received healthy response to inference request in 1.941023349761963s
Received healthy response to inference request in 9.567286491394043s
Received healthy response to inference request in 9.106594324111938s
Received healthy response to inference request in 2.1239192485809326s
2026-03-11T07:57:57.457085+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
Received healthy response to inference request in 2.0416433811187744s
{"detail":"('http://chaiml-pony-d3-g46-o4-l-75766-v2-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination')"}
Received unhealthy response to inference request!
Received healthy response to inference request in 1.9726693630218506s
Received healthy response to inference request in 1.9585750102996826s
Received healthy response to inference request in 1.9530208110809326s
Received healthy response to inference request in 1.952397108078003s
Received healthy response to inference request in 2.1591408252716064s
Received healthy response to inference request in 1.9754130840301514s
Received healthy response to inference request in 2.028813362121582s
Received healthy response to inference request in 2.0671842098236084s
Received healthy response to inference request in 1.9869422912597656s
30 requests
1 failed requests
5th percentile: 1.9438667058944703
10th percentile: 1.951891589164734
20th percentile: 1.960967206954956
30th percentile: 1.9722837686538697
40th percentile: 1.983394241333008
50th percentile: 2.029066324234009
60th percentile: 2.0425617694854736
70th percentile: 2.063571310043335
80th percentile: 2.190781211853028
90th percentile: 8.993442010879518
95th percentile: 9.1133518576622
99th percentile: 9.437248826026917
mean time: 3.113521679242452
%s, retrying in %s seconds...
Received healthy response to inference request in 1.939204454421997s
Received healthy response to inference request in 1.9698259830474854s
Received healthy response to inference request in 1.9615602493286133s
Received healthy response to inference request in 2.140023946762085s
Received healthy response to inference request in 2.0232977867126465s
Received healthy response to inference request in 2.1054039001464844s
Received healthy response to inference request in 2.0452823638916016s
Received healthy response to inference request in 1.9812803268432617s
Received healthy response to inference request in 2.251009702682495s
Received healthy response to inference request in 1.947187900543213s
Received healthy response to inference request in 1.9384698867797852s
Received healthy response to inference request in 1.9457204341888428s
Received healthy response to inference request in 1.993166446685791s
Received healthy response to inference request in 2.013648509979248s
Received healthy response to inference request in 2.0385935306549072s
Received healthy response to inference request in 2.1030266284942627s
2026-03-11T07:58:57.556224+00:00 monitor updated for chaiml-pony-d3-g46-o4-l_75766_v2
Received healthy response to inference request in 1.958678960800171s
Received healthy response to inference request in 1.936622142791748s
Received healthy response to inference request in 2.12796950340271s
Received healthy response to inference request in 1.9561452865600586s
Received healthy response to inference request in 1.9620332717895508s
Received healthy response to inference request in 2.0963211059570312s
Received healthy response to inference request in 1.988600254058838s
Received healthy response to inference request in 2.0325582027435303s
Received healthy response to inference request in 2.1216049194335938s
Received healthy response to inference request in 1.9573187828063965s
Received healthy response to inference request in 2.023261070251465s
Received healthy response to inference request in 2.1917166709899902s
Received healthy response to inference request in 2.2301955223083496s
Received healthy response to inference request in 2.190203905105591s
30 requests
0 failed requests
5th percentile: 1.9388004422187806
10th percentile: 1.9450688362121582
20th percentile: 1.9570840835571288
30th percentile: 1.9618913650512695
40th percentile: 1.9856722831726075
50th percentile: 2.0184547901153564
60th percentile: 2.034972333908081
70th percentile: 2.0983327627182007
80th percentile: 2.122877836227417
90th percentile: 2.1903551816940308
95th percentile: 2.212880039215088
99th percentile: 2.244973590373993
mean time: 2.0389977216720583
Pipeline stage StressChecker completed in 160.48s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.70s
Shutdown handler de-registered
chaiml-pony-d3-g46-o4-l_75766_v2 status is now deployed due to DeploymentManager action
chaiml-pony-d3-g46-o4-l_75766_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3-g46-o4-l_75766_v2 status is now torndown due to DeploymentManager action