developer_uid: zonemercy
submission_id: chaiml-pony-v2-g46-lr1_80834_v25
model_name: chaiml-pony-v2-g46-lr1_80834_v25
model_group: ChaiML/pony-v2-g46-lr1e4
status: torndown
timestamp: 2026-03-08T03:52:20+00:00
num_battles: 12180
num_wins: 6854
celo_rating: 1338.97
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v2-g46-lr1_80834_v25
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-04
win_ratio: 0.5627257799671592
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '[gMASK]<sop><|system|>\nRespond as a high quality storyteller.<|user|>\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-g46-lr1-80834-v25-uploader
Waiting for job on chaiml-pony-v2-g46-lr1-80834-v25-uploader to finish
chaiml-pony-v2-g46-lr1-80834-v25-uploader: Using quantization_mode: fp8
chaiml-pony-v2-g46-lr1-80834-v25-uploader: Checking if ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-pony-v2-g46-lr1-80834-v25-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-v2-g46-lr1-80834-v25-uploader: Downloading snapshot of ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8...
chaiml-pony-v2-g46-lr1-80834-v25-uploader: Downloaded in 103.182s
chaiml-pony-v2-g46-lr1-80834-v25-uploader: Processed model ChaiML/pony-v2-g46-lr1e4ep1r64b16 in 106.730s
chaiml-pony-v2-g46-lr1-80834-v25-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1-80834-v25-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v25-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-g46-lr1-80834-v25-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-g46-lr1-80834-v25-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-g46-lr1-80834-v25-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v25-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v25-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v25-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v25-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v25-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v25-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v25-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v25-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-g46-lr1-80834-v25-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v25-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-g46-lr1-80834-v25-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-g46-lr1-80834-v25-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-g46-lr1-80834-v25-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/.gitattributes
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/special_tokens_map.json
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/generation_config.json
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/config.json
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/chat_template.jinja
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model.safetensors.index.json
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/tokenizer_config.json
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/recipe.yaml
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/tokenizer.json
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00072-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00071-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00043-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00019-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00014-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00005-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00009-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00021-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00052-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00013-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00029-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00064-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00023-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00065-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00061-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00046-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00068-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00070-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00018-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00017-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00047-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00049-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00060-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00051-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00048-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00015-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00001-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00031-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00069-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00034-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00039-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00030-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00056-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00008-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00053-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00012-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00028-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00066-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00007-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00062-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00036-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00025-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00003-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00067-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00033-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00038-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00063-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00011-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00026-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00004-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00006-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00002-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00057-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00055-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00020-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00022-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00045-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00027-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00041-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00054-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00040-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00010-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00044-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00024-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00035-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00016-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00058-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00032-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00059-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00042-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00050-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v25-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v25/default/model-00037-of-00072.safetensors
Job chaiml-pony-v2-g46-lr1-80834-v25-uploader completed after 249.2s with status: succeeded
Stopping job with name chaiml-pony-v2-g46-lr1-80834-v25-uploader
Pipeline stage VLLMUploader completed in 249.72s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.27s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v2-g46-lr1-80834-v25
Waiting for inference service chaiml-pony-v2-g46-lr1-80834-v25 to be ready
Inference service chaiml-pony-v2-g46-lr1-80834-v25 ready after 611.3651578426361s
Pipeline stage VLLMDeployer completed in 612.69s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.115896224975586s
Received healthy response to inference request in 2.53975772857666s
Received healthy response to inference request in 2.0501554012298584s
Received healthy response to inference request in 2.159942150115967s
Received healthy response to inference request in 2.0340800285339355s
Received healthy response to inference request in 2.4309332370758057s
Received healthy response to inference request in 2.049412727355957s
Received healthy response to inference request in 2.0518641471862793s
Received healthy response to inference request in 2.025846242904663s
Received healthy response to inference request in 2.090834617614746s
Received healthy response to inference request in 2.0298023223876953s
Received healthy response to inference request in 2.1273672580718994s
Received healthy response to inference request in 2.3980321884155273s
Received healthy response to inference request in 2.0513694286346436s
Received healthy response to inference request in 2.319998025894165s
Received healthy response to inference request in 2.065330743789673s
Received healthy response to inference request in 2.0281577110290527s
Received healthy response to inference request in 2.0656955242156982s
Received healthy response to inference request in 2.571152448654175s
Received healthy response to inference request in 2.0406243801116943s
Received healthy response to inference request in 2.5129265785217285s
Received healthy response to inference request in 2.2925846576690674s
Received healthy response to inference request in 2.6215221881866455s
Received healthy response to inference request in 2.1196045875549316s
Received healthy response to inference request in 2.0812859535217285s
Received healthy response to inference request in 2.1210293769836426s
Received healthy response to inference request in 2.1908538341522217s
Received healthy response to inference request in 2.3596982955932617s
Received healthy response to inference request in 2.378183603286743s
Received healthy response to inference request in 2.3024256229400635s
30 requests
0 failed requests
5th percentile: 2.028897786140442
10th percentile: 2.0336522579193117
20th percentile: 2.050006866455078
30th percentile: 2.0612907648086547
40th percentile: 2.087015151977539
50th percentile: 2.120316982269287
60th percentile: 2.1723068237304686
70th percentile: 2.3076973438262938
80th percentile: 2.3821533203125003
90th percentile: 2.515609693527222
95th percentile: 2.557024824619293
99th percentile: 2.606914963722229
mean time: 2.2075455745061237
Pipeline stage StressChecker completed in 69.63s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.81s
Shutdown handler de-registered
chaiml-pony-v2-g46-lr1_80834_v25 status is now deployed due to DeploymentManager action
chaiml-pony-v2-g46-lr1_80834_v25 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v2-g46-lr1_80834_v25 status is now torndown due to DeploymentManager action