developer_uid: zonemercy
submission_id: chaiml-pony-v2-g46-lr1_80834_v29
model_name: chaiml-pony-v2-g46-lr1_80834_v29
model_group: ChaiML/pony-v2-g46-lr1e4
status: torndown
timestamp: 2026-03-08T21:03:58+00:00
num_battles: 12006
num_wins: 6786
celo_rating: 1344.37
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v2-g46-lr1_80834_v29
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-05
win_ratio: 0.5652173913043478
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-g46-lr1-80834-v29-uploader
Waiting for job on chaiml-pony-v2-g46-lr1-80834-v29-uploader to finish
chaiml-pony-v2-g46-lr1-80834-v29-uploader: Using quantization_mode: fp8
chaiml-pony-v2-g46-lr1-80834-v29-uploader: Checking if ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-pony-v2-g46-lr1-80834-v29-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-v2-g46-lr1-80834-v29-uploader: Downloading snapshot of ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8...
chaiml-pony-v2-g46-lr1-80834-v29-uploader: Downloaded in 122.736s
chaiml-pony-v2-g46-lr1-80834-v29-uploader: Processed model ChaiML/pony-v2-g46-lr1e4ep1r64b16 in 126.250s
chaiml-pony-v2-g46-lr1-80834-v29-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1-80834-v29-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v29-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-g46-lr1-80834-v29-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-g46-lr1-80834-v29-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-g46-lr1-80834-v29-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v29-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v29-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v29-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v29-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v29-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v29-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v29-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v29-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-g46-lr1-80834-v29-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v29-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-g46-lr1-80834-v29-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-g46-lr1-80834-v29-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-g46-lr1-80834-v29-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/generation_config.json
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/config.json
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/chat_template.jinja
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/special_tokens_map.json
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/tokenizer_config.json
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/.gitattributes
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/recipe.yaml
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model.safetensors.index.json
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/tokenizer.json
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00072-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00071-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00007-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00047-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00001-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00058-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00025-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00049-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00069-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00011-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00061-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00029-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00064-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00036-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00043-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00032-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00005-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00009-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00056-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00030-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00020-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00052-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00028-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00035-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00017-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00026-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00010-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00063-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00059-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00002-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00037-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00003-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00048-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00044-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00024-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00019-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00057-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00034-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00051-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00004-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00066-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00018-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00054-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00053-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00038-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00042-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00062-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00055-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00008-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00070-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00015-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00045-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00013-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00060-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00046-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00050-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00033-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00006-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00021-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00022-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00065-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00027-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00068-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00041-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00039-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00040-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00031-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00023-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00012-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00016-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00067-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v29-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v29/default/model-00014-of-00072.safetensors
Job chaiml-pony-v2-g46-lr1-80834-v29-uploader completed after 267.92s with status: succeeded
Stopping job with name chaiml-pony-v2-g46-lr1-80834-v29-uploader
Pipeline stage VLLMUploader completed in 268.39s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.72s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v2-g46-lr1-80834-v29
Waiting for inference service chaiml-pony-v2-g46-lr1-80834-v29 to be ready
Inference service chaiml-pony-v2-g46-lr1-80834-v29 ready after 551.1632807254791s
Pipeline stage VLLMDeployer completed in 551.63s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2421634197235107s
Received healthy response to inference request in 2.1253502368927s
Received healthy response to inference request in 2.176555633544922s
Received healthy response to inference request in 2.342223882675171s
Received healthy response to inference request in 2.3089890480041504s
Received healthy response to inference request in 2.3328511714935303s
Received healthy response to inference request in 2.316599130630493s
Received healthy response to inference request in 2.1236486434936523s
Received healthy response to inference request in 2.2085983753204346s
Received healthy response to inference request in 2.3611538410186768s
Received healthy response to inference request in 2.823436975479126s
Received healthy response to inference request in 2.13835072517395s
Received healthy response to inference request in 2.5118987560272217s
Received healthy response to inference request in 2.1435389518737793s
Received healthy response to inference request in 2.127481460571289s
Received healthy response to inference request in 2.539006233215332s
Received healthy response to inference request in 2.550039052963257s
Received healthy response to inference request in 2.196990728378296s
Received healthy response to inference request in 2.4312891960144043s
Received healthy response to inference request in 2.1706767082214355s
Received healthy response to inference request in 2.291198492050171s
Received healthy response to inference request in 2.2007625102996826s
Received healthy response to inference request in 2.694110870361328s
Received healthy response to inference request in 2.0926408767700195s
Received healthy response to inference request in 2.129976749420166s
Received healthy response to inference request in 2.182913303375244s
Received healthy response to inference request in 2.7559492588043213s
Received healthy response to inference request in 2.032811403274536s
Received healthy response to inference request in 2.290354013442993s
Received healthy response to inference request in 2.3989434242248535s
30 requests
0 failed requests
5th percentile: 2.106594371795654
10th percentile: 2.1251800775527956
20th percentile: 2.1366759300231934
30th percentile: 2.174791955947876
40th percentile: 2.199253797531128
50th percentile: 2.266258716583252
60th percentile: 2.3120330810546874
70th percentile: 2.3479028701782227
80th percentile: 2.447411108016968
90th percentile: 2.5644462347030643
95th percentile: 2.728121984004974
99th percentile: 2.8038655376434325
mean time: 2.3080167690912883
Pipeline stage StressChecker completed in 73.50s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.78s
Shutdown handler de-registered
chaiml-pony-v2-g46-lr1_80834_v29 status is now deployed due to DeploymentManager action
chaiml-pony-v2-g46-lr1_80834_v29 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v2-g46-lr1_80834_v29 status is now torndown due to DeploymentManager action