chaiml-pony-v2-g46-lr1e_80834

developer_uid: chai_backend_admin
submission_id: chaiml-pony-v2-g46-lr1e_80834_v2
model_name: chaiml-pony-v2-g46-lr1e_80834_v2
model_group: ChaiML/pony-v2-g46-lr1e4
status: deployed
timestamp: 2026-03-02T20:05:45+00:00
num_battles: 6174
num_wins: 3552
celo_rating: 1347.32
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v2-g46-lr1e_80834_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v2-g46-lr1e4ep1r64b16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-02
win_ratio: 0.5753158406219631
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_end|>', '###', 'You:', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '[gMASK]<sop><|system|>\nRespond as a high quality storyteller.<|user|>\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-g46-lr1e-80834-v2-uploader
Waiting for job on chaiml-pony-v2-g46-lr1e-80834-v2-uploader to finish
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: Using quantization_mode: fp8
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: Checking if ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: Downloading snapshot of ChaiML/pony-v2-g46-lr1e4ep1r64b16-FP8...
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: Downloaded in 153.241s
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: Processed model ChaiML/pony-v2-g46-lr1e4ep1r64b16 in 156.539s
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-g46-lr1e-80834-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-g46-lr1e-80834-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-g46-lr1e-80834-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-g46-lr1e-80834-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/.gitattributes
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/generation_config.json
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/chat_template.jinja
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/config.json
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/tokenizer_config.json
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/recipe.yaml
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/special_tokens_map.json
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model.safetensors.index.json
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/tokenizer.json
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00072-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00071-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00019-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00034-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00013-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00037-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00009-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00062-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00011-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00025-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00022-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00003-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00024-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00053-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00055-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00026-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00043-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00016-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00032-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00052-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00014-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00041-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00039-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00065-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00018-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00042-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00028-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00035-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00005-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00023-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00006-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00027-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00066-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00067-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00046-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00036-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00004-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00068-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00045-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00002-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00057-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00070-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00020-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00001-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00054-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00017-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00033-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00008-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00010-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00058-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00038-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00044-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00030-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00040-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00012-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00050-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00064-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00048-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00059-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00015-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00049-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00051-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00061-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00047-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00021-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00060-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00063-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00007-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00031-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00069-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00056-of-00072.safetensors
chaiml-pony-v2-g46-lr1e-80834-v2-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-g46-lr1e-80834-v2/default/model-00029-of-00072.safetensors
Job chaiml-pony-v2-g46-lr1e-80834-v2-uploader completed after 390.75s with status: succeeded
Stopping job with name chaiml-pony-v2-g46-lr1e-80834-v2-uploader
Pipeline stage VLLMUploader completed in 392.02s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.09s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v2-g46-lr1e-80834-v2
Waiting for inference service chaiml-pony-v2-g46-lr1e-80834-v2 to be ready
Inference service chaiml-pony-v2-g46-lr1e-80834-v2 ready after 566.0621435642242s
Pipeline stage VLLMDeployer completed in 567.70s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.6653716564178467s
Received healthy response to inference request in 2.6578240394592285s
Received healthy response to inference request in 2.701408624649048s
Received healthy response to inference request in 2.1093227863311768s
Received healthy response to inference request in 2.1274123191833496s
Received healthy response to inference request in 2.228259563446045s
Received healthy response to inference request in 2.455165386199951s
Received healthy response to inference request in 2.2541275024414062s
Received healthy response to inference request in 2.2496724128723145s
Received healthy response to inference request in 2.1665430068969727s
Received healthy response to inference request in 2.248546838760376s
Received healthy response to inference request in 2.2205264568328857s
Received healthy response to inference request in 2.1412861347198486s
Received healthy response to inference request in 2.139549970626831s
Received healthy response to inference request in 2.4179606437683105s
Received healthy response to inference request in 2.4738240242004395s
Received healthy response to inference request in 2.453244924545288s
Received healthy response to inference request in 2.2978999614715576s
Received healthy response to inference request in 2.1835224628448486s
Received healthy response to inference request in 2.2153732776641846s
Received healthy response to inference request in 2.144237518310547s
Received healthy response to inference request in 2.4878952503204346s
Received healthy response to inference request in 2.4770827293395996s
Received healthy response to inference request in 2.1283013820648193s
Received healthy response to inference request in 2.6721079349517822s
Received healthy response to inference request in 2.5173757076263428s
Received healthy response to inference request in 2.308764934539795s
Received healthy response to inference request in 2.329461097717285s
Received healthy response to inference request in 2.226422071456909s
Received healthy response to inference request in 2.237708806991577s
30 requests
0 failed requests
5th percentile: 2.127812397480011
10th percentile: 2.13842511177063
20th percentile: 2.1620819091796877
30th percentile: 2.2189805030822756
40th percentile: 2.2339291095733644
50th percentile: 2.2518999576568604
60th percentile: 2.317043399810791
70th percentile: 2.453821063041687
80th percentile: 2.4792452335357664
90th percentile: 2.65857880115509
95th percentile: 2.669076609611511
99th percentile: 2.692911424636841
mean time: 2.3312066475550335
Pipeline stage StressChecker completed in 81.98s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.55s
Shutdown handler de-registered
chaiml-pony-v2-g46-lr1e_80834_v2 status is now deployed due to DeploymentManager action