developer_uid: zonemercy
submission_id: chaiml-pony-v2-g46-lr1_80834_v22
model_name: chaiml-pony-v2-g46-lr1_80834_v22
model_group: ChaiML/pony-v2-g46-lr1e4
status: torndown
timestamp: 2026-03-08T03:43:29+00:00
num_battles: 10585
num_wins: 5879
celo_rating: 1326.86
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v2-g46-lr1_80834_v22
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-04
win_ratio: 0.5554085970713274
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n<think></think>\n{message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-g46-lr1-80834-v22-uploader
Waiting for job on chaiml-pony-v2-g46-lr1-80834-v22-uploader to finish
chaiml-pony-v2-g46-lr1-80834-v22-uploader: Using quantization_mode: fp8
chaiml-pony-v2-g46-lr1-80834-v22-uploader: Checking if ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-pony-v2-g46-lr1-80834-v22-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-v2-g46-lr1-80834-v22-uploader: Downloading snapshot of ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8...
chaiml-pony-v2-g46-lr1-80834-v22-uploader: Downloaded in 128.274s
chaiml-pony-v2-g46-lr1-80834-v22-uploader: Processed model ChaiML/pony-v2-g46-lr1e4ep1r64b16 in 131.704s
chaiml-pony-v2-g46-lr1-80834-v22-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1-80834-v22-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v22-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-g46-lr1-80834-v22-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-g46-lr1-80834-v22-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-g46-lr1-80834-v22-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v22-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v22-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v22-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v22-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v22-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v22-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1-80834-v22-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1-80834-v22-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-g46-lr1-80834-v22-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-g46-lr1-80834-v22-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-g46-lr1-80834-v22-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-g46-lr1-80834-v22-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-g46-lr1-80834-v22-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/generation_config.json
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/chat_template.jinja
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/config.json
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/.gitattributes
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/special_tokens_map.json
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/recipe.yaml
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/tokenizer_config.json
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model.safetensors.index.json
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/tokenizer.json
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00072-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00071-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00005-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00018-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00053-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00057-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00015-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00013-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00059-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00022-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00034-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00003-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00067-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00010-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00030-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00009-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00019-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00014-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00060-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00039-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00020-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00068-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00025-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00042-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00038-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00062-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00051-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00004-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00045-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00024-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00029-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00055-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00063-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00026-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00028-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00070-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00036-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00012-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00032-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00046-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00043-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00031-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00037-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00023-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00027-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00049-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00064-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00002-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00021-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00048-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00052-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00058-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00011-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00001-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00035-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00069-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00017-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00041-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00066-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00061-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00065-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00054-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00016-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00033-of-00072.safetensors
chaiml-pony-v2-g46-lr1-80834-v22-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1-80834-v22/default/model-00044-of-00072.safetensors
Job chaiml-pony-v2-g46-lr1-80834-v22-uploader completed after 288.62s with status: succeeded
Stopping job with name chaiml-pony-v2-g46-lr1-80834-v22-uploader
Pipeline stage VLLMUploader completed in 289.13s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.71s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v2-g46-lr1-80834-v22
Waiting for inference service chaiml-pony-v2-g46-lr1-80834-v22 to be ready
Inference service chaiml-pony-v2-g46-lr1-80834-v22 ready after 631.7852356433868s
Pipeline stage VLLMDeployer completed in 632.23s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1921188831329346s
Received healthy response to inference request in 2.284677505493164s
Received healthy response to inference request in 2.009995222091675s
Received healthy response to inference request in 2.506751537322998s
Received healthy response to inference request in 2.1616549491882324s
Received healthy response to inference request in 2.2867448329925537s
Received healthy response to inference request in 1.9781711101531982s
Received healthy response to inference request in 2.1252858638763428s
Received healthy response to inference request in 2.205265522003174s
Received healthy response to inference request in 2.047502279281616s
Received healthy response to inference request in 2.146341323852539s
Received healthy response to inference request in 2.49609112739563s
Received healthy response to inference request in 1.9999253749847412s
Received healthy response to inference request in 2.1064834594726562s
Received healthy response to inference request in 2.5919058322906494s
Received healthy response to inference request in 2.1877336502075195s
Received healthy response to inference request in 2.355738639831543s
Received healthy response to inference request in 2.000274419784546s
Received healthy response to inference request in 2.131364107131958s
Received healthy response to inference request in 2.030217170715332s
Received healthy response to inference request in 2.2059671878814697s
Received healthy response to inference request in 2.025348663330078s
Received healthy response to inference request in 2.1274707317352295s
Received healthy response to inference request in 2.012153148651123s
Received healthy response to inference request in 2.0975284576416016s
Received healthy response to inference request in 2.1572906970977783s
Received healthy response to inference request in 2.06048846244812s
Received healthy response to inference request in 2.0232415199279785s
Received healthy response to inference request in 2.257537841796875s
Received healthy response to inference request in 2.054133176803589s
30 requests
0 failed requests
5th percentile: 2.0000824451446535
10th percentile: 2.009023141860962
20th percentile: 2.0249272346496583
30th percentile: 2.052143907546997
40th percentile: 2.1029014587402344
50th percentile: 2.1294174194335938
60th percentile: 2.15903639793396
70th percentile: 2.1960628747940065
80th percentile: 2.262965774536133
90th percentile: 2.369773888587952
95th percentile: 2.5019543528556825
99th percentile: 2.5672110867500306
mean time: 2.1621800899505614
Pipeline stage StressChecker completed in 70.35s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.72s
Shutdown handler de-registered
chaiml-pony-v2-g46-lr1_80834_v22 status is now deployed due to DeploymentManager action
chaiml-pony-v2-g46-lr1_80834_v22 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v2-g46-lr1_80834_v22 status is now torndown due to DeploymentManager action