developer_uid: zonemercy
submission_id: chaiml-pony-d3-g46-pv2-_64172_v1
model_name: chaiml-pony-d3-g46-pv2-_64172_v1
model_group: ChaiML/pony-d3-g46-pv2-l
status: torndown
timestamp: 2026-03-09T12:23:47+00:00
num_battles: 11960
num_wins: 6923
celo_rating: 1356.11
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-d3-g46-pv2-lr5e6ep2r64b8
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-d3-g46-pv2-_64172_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-d3-g46-pv2-lr5e6ep2r64b8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-06
win_ratio: 0.5788461538461539
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-d3-g46-pv2-64172-v1-uploader
Waiting for job on chaiml-pony-d3-g46-pv2-64172-v1-uploader to finish
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Using quantization_mode: fp8
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Checking if ChaiML/pony-d3-g46-pv2-lr5e6ep2r64b8-FP8 already exists in ChaiML
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Downloading snapshot of ChaiML/pony-d3-g46-pv2-lr5e6ep2r64b8...
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Downloaded in 265.012s
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Loading /tmp/model_input...
chaiml-pony-d3-g46-pv2-64172-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-pony-d3-g46-pv2-64172-v1-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Applying quantization...
chaiml-pony-d3-g46-pv2-64172-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-pony-d3-g46-pv2-64172-v1-uploader: 2026-03-06T01:20:26.278127-0800 | reset | INFO - Compression lifecycle reset
chaiml-pony-d3-g46-pv2-64172-v1-uploader: 2026-03-06T01:20:26.351418-0800 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-pony-d3-g46-pv2-64172-v1-uploader: 2026-03-06T01:21:23.653010-0800 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-pony-d3-g46-pv2-64172-v1-uploader: 2026-03-06T01:21:23.653519-0800 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-pony-d3-g46-pv2-64172-v1-uploader: 2026-03-06T01:28:35.646837-0800 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-pony-d3-g46-pv2-64172-v1-uploader: 2026-03-06T01:28:42.660234-0800 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Saving to /dev/shm/model_output...
chaiml-pony-d3-g46-pv2-64172-v1-uploader: 2026-03-06T01:28:42.695841-0800 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:10:34 (0:00:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 8/80 (28.2M/354.7G) | pre-uploaded: 0/1 (0.0/354.7G) (+72 unsure) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 72 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 53
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: 







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:11:34 (0:01:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 3/73 (5.1G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 70 | committing: 0 | waiting: 56
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:12:34 (0:02:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 11/73 (45.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 62 | committing: 0 | waiting: 64
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:13:34 (0:03:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 18/73 (80.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 55 | committing: 0 | waiting: 71
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:14:34 (0:04:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 19/73 (85.0G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 54 | committing: 0 | waiting: 72
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:15:34 (0:05:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 27/73 (124.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 46 | committing: 0 | waiting: 80
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:16:34 (0:06:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 35/73 (164.9G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 38 | committing: 0 | waiting: 88
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:17:34 (0:07:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:18:34 (0:08:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 43/73 (204.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 96
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:19:34 (0:09:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 52/73 (249.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 21 | committing: 0 | waiting: 105
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:20:34 (0:10:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 58/73 (279.8G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 15 | committing: 0 | waiting: 111
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
Failed to get response for submission function_jukol_2026-02-20: HTTPConnectionPool(host='chaiml-llama-1-8b-user-48416-v5-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Max retries exceeded with url: /classify (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c0c5a320c90>, 'Connection to chaiml-llama-1-8b-user-48416-v5-predictor.tenant-chaiml-guanaco.k2.chaiverse.com timed out. (connect timeout=12.0)'))
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:21:34 (0:11:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 66/73 (319.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 7 | committing: 0 | waiting: 119
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:22:34 (0:12:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:23:34 (0:13:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 0/80 (0.0/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              







chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------- 2026-03-06 02:24:34 (0:14:00) ----------
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Files:   hashed 80/80 (354.7G/354.7G) | pre-uploaded: 73/73 (354.7G/354.7G) | committed: 50/80 (204.8G/354.7G) | ignored: 0
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-pony-d3-g46-pv2-64172-v1-uploader: ---------------------------------------------------
chaiml-pony-d3-g46-pv2-64172-v1-uploader:                              Processed model ChaiML/pony-d3-g46-pv2-lr5e6ep2r64b8 in 4180.199s
chaiml-pony-d3-g46-pv2-64172-v1-uploader: creating bucket guanaco-vllm-models
chaiml-pony-d3-g46-pv2-64172-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-64172-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-d3-g46-pv2-64172-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-d3-g46-pv2-64172-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-d3-g46-pv2-64172-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-64172-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-pv2-64172-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-64172-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-d3-g46-pv2-64172-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-64172-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-pv2-64172-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-d3-g46-pv2-64172-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-d3-g46-pv2-64172-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-d3-g46-pv2-64172-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-d3-g46-pv2-64172-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-d3-g46-pv2-64172-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-d3-g46-pv2-64172-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-d3-g46-pv2-64172-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/generation_config.json
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/tokenizer_config.json
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model.safetensors.index.json
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00072-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00071-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00008-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00026-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00027-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00041-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00029-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00065-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00003-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00053-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00060-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00016-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00068-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00028-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00030-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00057-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00066-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00055-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00037-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00047-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00034-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00054-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00062-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00013-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00051-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00036-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00031-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00004-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00069-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00021-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00033-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00049-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00007-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00035-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00010-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00067-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00011-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00019-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00063-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00009-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00043-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00015-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00056-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00058-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00050-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00024-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00018-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00064-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00032-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00006-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00038-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00005-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00040-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00046-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00052-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00001-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00017-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00020-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00045-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00014-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00048-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00022-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00070-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00059-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00061-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00044-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00023-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00002-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00025-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00039-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00042-of-00072.safetensors
chaiml-pony-d3-g46-pv2-64172-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-pony-d3-g46-pv2-64172-v1/default/model-00012-of-00072.safetensors
Job chaiml-pony-d3-g46-pv2-64172-v1-uploader completed after 4335.62s with status: succeeded
Stopping job with name chaiml-pony-d3-g46-pv2-64172-v1-uploader
Pipeline stage VLLMUploader completed in 4336.32s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.83s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-d3-g46-pv2-64172-v1
Waiting for inference service chaiml-pony-d3-g46-pv2-64172-v1 to be ready
Inference service chaiml-pony-d3-g46-pv2-64172-v1 ready after 601.018147945404s
Pipeline stage VLLMDeployer completed in 601.52s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1329243183135986s
Received healthy response to inference request in 2.1004576683044434s
Received healthy response to inference request in 2.489248752593994s
Received healthy response to inference request in 2.119074583053589s
Received healthy response to inference request in 2.140957832336426s
Received healthy response to inference request in 2.1286582946777344s
Received healthy response to inference request in 2.089634656906128s
Received healthy response to inference request in 2.1686277389526367s
Received healthy response to inference request in 2.535569190979004s
Received healthy response to inference request in 2.191222667694092s
Received healthy response to inference request in 2.3814432621002197s
Received healthy response to inference request in 2.1533851623535156s
Received healthy response to inference request in 2.0661330223083496s
Received healthy response to inference request in 2.0925309658050537s
Received healthy response to inference request in 2.1425411701202393s
Received healthy response to inference request in 2.1808993816375732s
Received healthy response to inference request in 2.6017706394195557s
Received healthy response to inference request in 2.2253780364990234s
Received healthy response to inference request in 2.100132942199707s
Received healthy response to inference request in 2.1902318000793457s
Received healthy response to inference request in 2.332050323486328s
Received healthy response to inference request in 2.1152448654174805s
Received healthy response to inference request in 2.1855416297912598s
Received healthy response to inference request in 2.111180067062378s
Received healthy response to inference request in 2.2665061950683594s
Received healthy response to inference request in 2.0864675045013428s
Received healthy response to inference request in 2.425926685333252s
Received healthy response to inference request in 2.1159496307373047s
Received healthy response to inference request in 2.5308420658111572s
Received healthy response to inference request in 2.1228394508361816s
30 requests
0 failed requests
5th percentile: 2.0878927230834963
10th percentile: 2.092241334915161
20th percentile: 2.109035587310791
30th percentile: 2.1181370973587037
40th percentile: 2.131217908859253
50th percentile: 2.1479631662368774
60th percentile: 2.182756280899048
70th percentile: 2.2014692783355714
80th percentile: 2.3419289112091066
90th percentile: 2.4934080839157104
95th percentile: 2.533441984653473
99th percentile: 2.5825722193717957
mean time: 2.2174456834793093
Pipeline stage StressChecker completed in 70.56s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.83s
Shutdown handler de-registered
chaiml-pony-d3-g46-pv2-_64172_v1 status is now deployed due to DeploymentManager action
chaiml-pony-d3-g46-pv2-_64172_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-d3-g46-pv2-_64172_v1 status is now torndown due to DeploymentManager action