developer_uid: richhx
submission_id: chaiml-pony-d3-g46-pv2-_13283_v2
model_name: chaiml-pony-d3-g46-pv2-_13283_v2
model_group: ChaiML/pony-d3-g46-pv2-l
status: deployed
timestamp: 2026-03-27T18:02:36+00:00
num_battles: 9314
num_wins: 5432
celo_rating: 1355.06
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8-W4A16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3-g46-pv2-_13283_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8-W4A16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5832080738672966
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '<|assistant|>', '####', '</s>', '<|im_end|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3-g46-pv2-13283-v2-uploader
Waiting for job on chaiml-pony-d3-g46-pv2-13283-v2-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3-g46-pv2-l-7830-v2-uploader
Waiting for job on chaiml-pony-d3-g46-pv2-l-7830-v2-uploader to finish
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: {repo_id} is already quantized
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: Using quantization_mode: none
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: Downloading snapshot of ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8-FP8...
2026-03-27T17:52:36.489893+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
2026-03-27T17:52:38.311253+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
2026-03-27T17:53:36.793451+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
2026-03-27T17:53:38.623510+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
chaiml-pony-d3-g46-pv2-13283-v2-uploader: Downloaded in 80.608s
2026-03-27T17:54:37.099810+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: Downloaded in 135.875s
2026-03-27T17:54:38.919904+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
chaiml-pony-d3-g46-pv2-13283-v2-uploader: Processed model ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8-W4A16 in 149.466s
chaiml-pony-d3-g46-pv2-13283-v2-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3-g46-pv2-13283-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-13283-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3-g46-pv2-13283-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3-g46-pv2-13283-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3-g46-pv2-13283-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-13283-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-pv2-13283-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-13283-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-pv2-13283-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-13283-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-pv2-13283-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-13283-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-pv2-13283-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3-g46-pv2-13283-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3-g46-pv2-13283-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3-g46-pv2-13283-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3-g46-pv2-13283-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3-g46-pv2-13283-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/generation_config.json
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/chat_template.jinja
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/config.json
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/.gitattributes
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/quantization_config.json
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/tokenizer_config.json
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/tokenizer.json
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model.safetensors.index.json
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00038-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00037-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00036-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00007-of-00038.safetensors
2026-03-27T17:55:37.404140+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
2026-03-27T17:55:39.195023+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00035-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00011-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00016-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00014-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00027-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00001-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00006-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00029-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00017-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00018-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00021-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00002-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00005-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00004-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00030-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00024-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00009-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00033-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00025-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00019-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00012-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00034-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00015-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00013-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00022-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00008-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00023-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00026-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00020-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00010-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00028-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00003-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00031-of-00038.safetensors
chaiml-pony-d3-g46-pv2-13283-v2-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-13283-v2/default/model-00032-of-00038.safetensors
Job chaiml-pony-d3-g46-pv2-13283-v2-uploader completed after 266.65s with status: succeeded
Stopping job with name chaiml-pony-d3-g46-pv2-13283-v2-uploader
Pipeline stage VLLMUploader completed in 268.27s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.43s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.38s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3-g46-pv2-13283-v2
Waiting for inference service chaiml-pony-d3-g46-pv2-13283-v2 to be ready
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: Processed model ChaiML/pony-d3-g46-pv2-lr5e6ep1r64b8-FP8 in 262.329s
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
2026-03-27T17:56:44.236689+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
2026-03-27T17:56:44.325319+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/.gitattributes
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/chat_template.jinja
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/config.json
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/generation_config.json
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00072-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00071-of-00072.safetensors
2026-03-27T17:57:44.699417+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
2026-03-27T17:57:44.811664+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00003-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00027-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00030-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00044-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00047-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00037-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00045-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00035-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00031-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00010-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00023-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00004-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00039-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00049-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00043-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00024-of-00072.safetensors
chaiml-pony-d3-g46-pv2-l-7830-v2-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-l-7830-v2/default/model-00021-of-00072.safetensors
Job chaiml-pony-d3-g46-pv2-l-7830-v2-uploader completed after 396.82s with status: succeeded
Stopping job with name chaiml-pony-d3-g46-pv2-l-7830-v2-uploader
Pipeline stage VLLMUploader completed in 398.77s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.46s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.36s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3-g46-pv2-l-7830-v2
Waiting for inference service chaiml-pony-d3-g46-pv2-l-7830-v2 to be ready
2026-03-27T17:58:45.109808+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
2026-03-27T17:58:45.228509+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
2026-03-27T17:59:45.524795+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
2026-03-27T17:59:45.666743+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
Inference service chaiml-pony-d3-g46-pv2-13283-v2 ready after 264.34930086135864s
Pipeline stage VLLMDeployer completed in 266.21s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.776982307434082s
2026-03-27T18:00:46.043552+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
2026-03-27T18:00:45.916113+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
Received healthy response to inference request in 8.924762964248657s
Received healthy response to inference request in 8.77453351020813s
Received healthy response to inference request in 2.0980658531188965s
Received healthy response to inference request in 8.901828289031982s
Received healthy response to inference request in 2.121722459793091s
Received healthy response to inference request in 2.10976243019104s
Received healthy response to inference request in 9.045119524002075s
Received healthy response to inference request in 2.3254706859588623s
Received healthy response to inference request in 2.185009479522705s
Received healthy response to inference request in 2.1378841400146484s
Received healthy response to inference request in 2.1490557193756104s
Received healthy response to inference request in 2.1174306869506836s
Received healthy response to inference request in 2.0651040077209473s
2026-03-27T18:01:48.873548+00:00 monitor updated for chaiml-pony-d3-g46-pv2-l_7830_v2
2026-03-27T18:01:49.031063+00:00 monitor updated for chaiml-pony-d3-g46-pv2-_13283_v2
Received healthy response to inference request in 2.0809617042541504s
Received healthy response to inference request in 2.1797070503234863s
Received healthy response to inference request in 2.309887647628784s
Received healthy response to inference request in 2.0939862728118896s
Received healthy response to inference request in 2.0891551971435547s
Received healthy response to inference request in 2.1326324939727783s
Received healthy response to inference request in 2.10388445854187s
Received healthy response to inference request in 2.110459089279175s
Received healthy response to inference request in 2.101132392883301s
Received healthy response to inference request in 2.258213996887207s
Received healthy response to inference request in 2.270087242126465s
Received healthy response to inference request in 2.6971116065979004s
Received healthy response to inference request in 2.4207780361175537s
Received healthy response to inference request in 2.113703727722168s
Received healthy response to inference request in 4.383466482162476s
Received healthy response to inference request in 2.212412118911743s
30 requests
0 failed requests
5th percentile: 2.084648776054382
10th percentile: 2.093503165245056
20th percentile: 2.1033340454101563
30th percentile: 2.11273033618927
40th percentile: 2.1282684803009033
50th percentile: 2.1643813848495483
60th percentile: 2.2307328701019284
70th percentile: 2.3145625591278076
80th percentile: 3.0343825817108203
90th percentile: 8.789466905593873
95th percentile: 8.914442360401154
99th percentile: 9.010216121673585
mean time: 3.3763437191645305
Pipeline stage StressChecker completed in 118.35s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.91s
Shutdown handler de-registered
chaiml-pony-d3-g46-pv2-_13283_v2 status is now deployed due to DeploymentManager action