developer_uid: zonemercy
submission_id: chaiml-pony-v2-g46-lr1_80834_v27
model_name: chaiml-pony-v2-g46-lr1_80834_v27
model_group: ChaiML/pony-v2-g46-lr1e4
status: torndown
timestamp: 2026-03-08T14:33:18+00:00
num_battles: 10650
num_wins: 6123
celo_rating: 1360.12
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v2-g46-lr1_80834_v27
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-05
win_ratio: 0.5749295774647887
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-g46-lr1-80834-v27-uploader
Waiting for job on chaiml-pony-v2-g46-lr1-80834-v27-uploader to finish
chaiml-pony-v2-g46-lr1-80834-v27-uploader: Using quantization_mode: fp8
chaiml-pony-v2-g46-lr1-80834-v27-uploader: Checking if ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-pony-v2-g46-lr1-80834-v27-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-v2-g46-lr1-80834-v27-uploader: Downloading snapshot of ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8...
chaiml-pony-v2-g46-lr1-80834-v27-uploader: Downloaded in 388.713s
chaiml-pony-v2-g46-lr1-80834-v27-uploader: Processed model ChaiML/pony-v2-g46-lr1e4ep1r64b16 in 392.245s
chaiml-pony-v2-g46-lr1-80834-v27-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1-80834-v27-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v27-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-g46-lr1-80834-v27-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-g46-lr1-80834-v27-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-g46-lr1-80834-v27-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v27-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v27-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v27-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v27-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v27-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v27-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v27-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v27-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-g46-lr1-80834-v27-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v27-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-g46-lr1-80834-v27-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-g46-lr1-80834-v27-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-g46-lr1-80834-v27-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/config.json
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/.gitattributes
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/generation_config.json
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/chat_template.jinja
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/tokenizer.json
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model.safetensors.index.json
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00072-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00071-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00009-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00003-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00025-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00031-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00010-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00021-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00023-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00026-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00048-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00024-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00016-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00027-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00032-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00068-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00022-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00056-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00014-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00061-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00066-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00043-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00002-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00057-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00064-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00034-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00052-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00015-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00012-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00053-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00007-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00051-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00055-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00030-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00046-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00067-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00011-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00040-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00062-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00033-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00069-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00041-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00070-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00044-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00059-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00006-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00037-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00050-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00013-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00008-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00001-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00035-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00054-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00018-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00058-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00020-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00042-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00017-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00049-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00036-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00047-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00029-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00004-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00038-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00028-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00063-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00019-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00060-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00065-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00005-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00045-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v27-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v27/default/model-00039-of-00072.safetensors
Job chaiml-pony-v2-g46-lr1-80834-v27-uploader completed after 533.78s with status: succeeded
Stopping job with name chaiml-pony-v2-g46-lr1-80834-v27-uploader
Pipeline stage VLLMUploader completed in 534.80s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.84s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v2-g46-lr1-80834-v27
Waiting for inference service chaiml-pony-v2-g46-lr1-80834-v27 to be ready
Inference service chaiml-pony-v2-g46-lr1-80834-v27 ready after 571.3303899765015s
Pipeline stage VLLMDeployer completed in 571.84s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2788608074188232s
Received healthy response to inference request in 2.176525592803955s
Received healthy response to inference request in 2.1260337829589844s
Received healthy response to inference request in 2.2930586338043213s
Received healthy response to inference request in 2.308424711227417s
Received healthy response to inference request in 2.027890205383301s
Received healthy response to inference request in 2.2788538932800293s
Received healthy response to inference request in 2.1227667331695557s
Received healthy response to inference request in 2.2066216468811035s
Received healthy response to inference request in 2.3716118335723877s
Received healthy response to inference request in 2.1180572509765625s
Received healthy response to inference request in 2.036403179168701s
Received healthy response to inference request in 2.142976760864258s
Received healthy response to inference request in 2.352099657058716s
Received healthy response to inference request in 2.1519851684570312s
Received healthy response to inference request in 3.0917561054229736s
Received healthy response to inference request in 2.196161985397339s
Received healthy response to inference request in 2.520127058029175s
Received healthy response to inference request in 2.079617977142334s
Received healthy response to inference request in 2.368574380874634s
Received healthy response to inference request in 2.3670501708984375s
Received healthy response to inference request in 2.1872570514678955s
Received healthy response to inference request in 2.457855701446533s
Received healthy response to inference request in 2.078598976135254s
Received healthy response to inference request in 2.3755035400390625s
Received healthy response to inference request in 2.888643741607666s
Received healthy response to inference request in 2.2132067680358887s
Received healthy response to inference request in 2.0349037647247314s
Received healthy response to inference request in 2.275566816329956s
Received healthy response to inference request in 2.5363104343414307s
30 requests
0 failed requests
5th percentile: 2.035578501224518
10th percentile: 2.074379396438599
20th percentile: 2.121824836730957
30th percentile: 2.1492826461791994
40th percentile: 2.1926000118255615
50th percentile: 2.2443867921829224
60th percentile: 2.2845399379730225
70th percentile: 2.3565848112106322
80th percentile: 2.3723901748657226
90th percentile: 2.5217453956604006
95th percentile: 2.7300937533378593
99th percentile: 3.0328535199165345
mean time: 2.2887768109639484
Pipeline stage StressChecker completed in 72.12s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.08s
Shutdown handler de-registered
chaiml-pony-v2-g46-lr1_80834_v27 status is now deployed due to DeploymentManager action
chaiml-pony-v2-g46-lr1_80834_v27 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v2-g46-lr1_80834_v27 status is now torndown due to DeploymentManager action