developer_uid: zonemercy
submission_id: chaiml-pony-v2-g46-lr1_80834_v28
model_name: chaiml-pony-v2-g46-lr1_80834_v28
model_group: ChaiML/pony-v2-g46-lr1e4
status: torndown
timestamp: 2026-03-08T20:52:23+00:00
num_battles: 10585
num_wins: 5906
celo_rating: 1359.7
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v2-g46-lr1_80834_v28
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-05
win_ratio: 0.5579593764761455
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n<think></think>\n{message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-g46-lr1-80834-v28-uploader
Waiting for job on chaiml-pony-v2-g46-lr1-80834-v28-uploader to finish
chaiml-pony-v2-g46-lr1-80834-v28-uploader: Using quantization_mode: fp8
chaiml-pony-v2-g46-lr1-80834-v28-uploader: Checking if ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-pony-v2-g46-lr1-80834-v28-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-v2-g46-lr1-80834-v28-uploader: Downloading snapshot of ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8...
chaiml-pony-v2-g46-lr1-80834-v28-uploader: Downloaded in 163.331s
chaiml-pony-v2-g46-lr1-80834-v28-uploader: Processed model ChaiML/pony-v2-g46-lr1e4ep1r64b16 in 166.806s
chaiml-pony-v2-g46-lr1-80834-v28-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1-80834-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v28-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-g46-lr1-80834-v28-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-g46-lr1-80834-v28-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-g46-lr1-80834-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v28-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v28-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v28-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v28-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v28-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v28-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-g46-lr1-80834-v28-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v28-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-g46-lr1-80834-v28-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-g46-lr1-80834-v28-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-g46-lr1-80834-v28-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/.gitattributes
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/config.json
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/chat_template.jinja
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/generation_config.json
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/special_tokens_map.json
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/recipe.yaml
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/tokenizer_config.json
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model.safetensors.index.json
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/tokenizer.json
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00072-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00071-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00042-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00032-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00028-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00019-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00055-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00017-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00003-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00013-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00020-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00010-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00025-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00044-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00041-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00007-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00009-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00021-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00022-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00069-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00015-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00002-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00031-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00024-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00006-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00016-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00001-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00052-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00068-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00051-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00047-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00050-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00005-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00027-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00040-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00038-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00058-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00012-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00067-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00062-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00033-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00056-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00049-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00045-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00035-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00018-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00030-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00054-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00008-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00004-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00026-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00066-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00053-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00060-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00029-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00048-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00034-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00057-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00039-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00046-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00059-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00064-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00011-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00036-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00037-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00070-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00065-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00014-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00043-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00063-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v28-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v28/default/model-00061-of-00072.safetensors
Job chaiml-pony-v2-g46-lr1-80834-v28-uploader completed after 310.02s with status: succeeded
Stopping job with name chaiml-pony-v2-g46-lr1-80834-v28-uploader
Pipeline stage VLLMUploader completed in 310.59s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.22s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v2-g46-lr1-80834-v28
Waiting for inference service chaiml-pony-v2-g46-lr1-80834-v28 to be ready
Inference service chaiml-pony-v2-g46-lr1-80834-v28 ready after 552.1052906513214s
Pipeline stage VLLMDeployer completed in 552.56s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.272986650466919s
Received healthy response to inference request in 2.3340792655944824s
Received healthy response to inference request in 2.8359670639038086s
Received healthy response to inference request in 2.626207113265991s
Received healthy response to inference request in 2.2633371353149414s
Received healthy response to inference request in 2.3778746128082275s
Received healthy response to inference request in 2.1038167476654053s
Received healthy response to inference request in 2.0474112033843994s
Received healthy response to inference request in 2.313701629638672s
Received healthy response to inference request in 2.4546878337860107s
Received healthy response to inference request in 2.093641519546509s
Received healthy response to inference request in 2.630254030227661s
Received healthy response to inference request in 2.335449695587158s
Received healthy response to inference request in 2.2958903312683105s
Received healthy response to inference request in 2.2411611080169678s
Received healthy response to inference request in 2.1928467750549316s
Received healthy response to inference request in 2.54758882522583s
Received healthy response to inference request in 2.8940136432647705s
Received healthy response to inference request in 2.3667049407958984s
Received healthy response to inference request in 2.1462159156799316s
Received healthy response to inference request in 2.1568288803100586s
Received healthy response to inference request in 2.500772476196289s
Received healthy response to inference request in 2.2321863174438477s
Received healthy response to inference request in 2.3128559589385986s
Received healthy response to inference request in 2.0235490798950195s
Received healthy response to inference request in 2.0352182388305664s
Received healthy response to inference request in 2.625441312789917s
Received healthy response to inference request in 2.0452816486358643s
Received healthy response to inference request in 2.11368727684021s
Received healthy response to inference request in 2.1023755073547363s
30 requests
0 failed requests
5th percentile: 2.0397467732429506
10th percentile: 2.047198247909546
20th percentile: 2.1035284996032715
30th percentile: 2.1536449909210207
40th percentile: 2.23757119178772
50th percentile: 2.3043731451034546
60th percentile: 2.3346274375915526
70th percentile: 2.400918579101562
80th percentile: 2.5631593227386475
90th percentile: 2.650825333595276
95th percentile: 2.8678926825523376
99th percentile: 3.163084478378296
mean time: 2.3507344245910646
Pipeline stage StressChecker completed in 76.76s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.83s
Shutdown handler de-registered
chaiml-pony-v2-g46-lr1_80834_v28 status is now deployed due to DeploymentManager action
chaiml-pony-v2-g46-lr1_80834_v28 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v2-g46-lr1_80834_v28 status is now torndown due to DeploymentManager action