developer_uid: zonemercy
submission_id: chaiml-pony-v3a-g47-lr1_18022_v5
model_name: chaiml-pony-v3a-g47-lr1_18022_v5
model_group: ChaiML/pony-v3a-g47-lr1e
status: torndown
timestamp: 2026-04-10T23:22:21+00:00
num_battles: 10289
num_wins: 5134
celo_rating: 1301.47
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v3a-g47-lr1e5ep1r64b32
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v3a-g47-lr1_18022_v5
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v3a-g47-lr1e5ep1r64b32
model_size: 24B
ranking_group: single
us_pacific_date: 2026-04-07
win_ratio: 0.49897949266206626
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|assistant|>', '<|im_end|>', '####', '<|user|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{bot_name}: {message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v3a-g47-lr1-18022-v5-uploader
Waiting for job on chaiml-pony-v3a-g47-lr1-18022-v5-uploader to finish
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: Using quantization_mode: w4a16
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: Checking if ChaiML/pony-v3a-g47-lr1e5ep1r64b32-W4A16 already exists in ChaiML
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: Model already exists. Downloading to /tmp/model_output...
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: Downloading snapshot of ChaiML/pony-v3a-g47-lr1e5ep1r64b32-W4A16...
2026-04-07T15:34:27.970073+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:35:28.423498+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: Downloaded in 79.939s
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: Processed model ChaiML/pony-v3a-g47-lr1e5ep1r64b32 in 82.586s
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3a-g47-lr1-18022-v5-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v3a-g47-lr1-18022-v5-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3a-g47-lr1-18022-v5-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3a-g47-lr1-18022-v5-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3a-g47-lr1-18022-v5-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v3a-g47-lr1-18022-v5-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v3a-g47-lr1-18022-v5-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v3a-g47-lr1-18022-v5-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: uploading /tmp/model_output to s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/config.json
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/.gitattributes
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/generation_config.json
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/quantization_config.json
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/chat_template.jinja
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/tokenizer_config.json
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model.safetensors.index.json
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/tokenizer.json
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00038-of-00038.safetensors
2026-04-07T15:36:28.566287+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00036-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00037-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00012-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00001-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00002-of-00038.safetensors
2026-04-07T15:37:28.678382+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Retrying (%r) after connection broken by '%r': %s
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00006-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00027-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00022-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00023-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00028-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00011-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00019-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00008-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00021-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00024-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00032-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00017-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00029-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00005-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00026-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00010-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00014-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00030-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00015-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00004-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00018-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00020-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00003-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00031-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00009-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00035-of-00038.safetensors
chaiml-pony-v3a-g47-lr1-18022-v5-uploader: cp /tmp/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-pony-v3a-g47-lr1-18022-v5/default/model-00013-of-00038.safetensors
Job chaiml-pony-v3a-g47-lr1-18022-v5-uploader completed after 267.75s with status: succeeded
Stopping job with name chaiml-pony-v3a-g47-lr1-18022-v5-uploader
Pipeline stage VLLMUploader completed in 268.43s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.13s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.80s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v3a-g47-lr1-18022-v5
Waiting for inference service chaiml-pony-v3a-g47-lr1-18022-v5 to be ready
2026-04-07T15:38:28.773801+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-07T15:39:28.871437+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:40:28.966370+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-07T15:41:36.157416+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:42:36.332561+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:43:36.474011+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:44:36.574462+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:45:36.737451+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:46:36.827803+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Retrying (%r) after connection broken by '%r': %s
2026-04-07T15:47:36.928364+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:48:37.069286+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:49:37.164685+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:50:37.262730+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Failed to get response for submission chaiml-gspo-glm47-kimi-_50225_v1: HTTPConnectionPool(host='chaiml-gspo-glm47-kimi-50225-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
2026-04-07T15:51:37.354057+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:52:37.444486+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:53:37.539291+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:54:37.632445+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:55:37.732951+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:56:37.824098+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:57:37.918935+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:58:38.011034+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T15:59:38.109344+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T16:00:38.206310+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T16:01:38.304769+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Failed to get response for submission chaiml-gspo-glm47-kimi-_82984_v1: ('http://chaiml-gspo-glm47-kimi-82984-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'activator request timeout')
2026-04-07T16:02:38.433774+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T16:03:38.531158+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Failed to get response for submission chaiml-gspo-glm47-kimi-_50225_v1: ('http://chaiml-gspo-glm47-kimi-50225-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'activator request timeout')
2026-04-07T16:04:38.637447+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T16:05:38.736567+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Failed to get response for submission chaiml-gspo-glm47-kimi-_82984_v1: ('http://chaiml-gspo-glm47-kimi-82984-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'activator request timeout')
Retrying (%r) after connection broken by '%r': %s
2026-04-07T16:06:38.846456+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T16:07:38.937653+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission chaiml-mega-v2-top2-g47_10262_v2: ('http://chaiml-mega-v2-top2-g47-10262-v2-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'request timeout')
2026-04-07T16:08:39.027156+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
2026-04-07T16:09:39.129870+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-07T16:10:39.237343+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Failed to get response for submission chaiml-mega-v2-top2-g47_10262_v2: HTTPConnectionPool(host='chaiml-mega-v2-top2-g47-10262-v2-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service chaiml-pony-v3a-g47-lr1-18022-v5 ready after 1987.3458273410797s
Pipeline stage VLLMDeployer completed in 1987.93s
run pipeline stage %s
Running pipeline stage StressChecker
upstream connect error or disconnect/reset before headers. reset reason: connection termination
Received unhealthy response to inference request!
Received healthy response to inference request in 8.701327323913574s
Received healthy response to inference request in 2.4541027545928955s
Received healthy response to inference request in 1.9705519676208496s
Received healthy response to inference request in 2.150402784347534s
Received healthy response to inference request in 2.1212167739868164s
Received healthy response to inference request in 2.189378261566162s
Received healthy response to inference request in 2.127976417541504s
Received healthy response to inference request in 2.193707227706909s
Received healthy response to inference request in 2.1371042728424072s
Received healthy response to inference request in 2.199598550796509s
2026-04-07T16:11:39.341090+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Received healthy response to inference request in 2.033893346786499s
Received healthy response to inference request in 2.252484083175659s
Received healthy response to inference request in 2.096208333969116s
Received healthy response to inference request in 2.7757070064544678s
Received healthy response to inference request in 2.1297690868377686s
Received healthy response to inference request in 2.094825029373169s
Received healthy response to inference request in 2.4216220378875732s
Received healthy response to inference request in 2.5980172157287598s
Received healthy response to inference request in 2.2126410007476807s
Received healthy response to inference request in 2.0692782402038574s
Received healthy response to inference request in 2.075068950653076s
Received healthy response to inference request in 1.928370475769043s
Received healthy response to inference request in 2.111711025238037s
Received healthy response to inference request in 2.3707289695739746s
Received healthy response to inference request in 2.013000726699829s
Received healthy response to inference request in 2.0162734985351562s
Received healthy response to inference request in 2.201448678970337s
Received healthy response to inference request in 2.136298418045044s
Received healthy response to inference request in 2.028791904449463s
30 requests
1 failed requests
5th percentile: 1.947352147102356
10th percentile: 2.008755850791931
20th percentile: 2.0328730583190917
30th percentile: 2.088898205757141
40th percentile: 2.1174144744873047
50th percentile: 2.1330337524414062
60th percentile: 2.1659929752349854
70th percentile: 2.2001535892486572
80th percentile: 2.2761330604553227
90th percentile: 2.4684942007064823
95th percentile: 2.6957466006278987
99th percentile: 6.982897431850438
mean time: 2.340322820345561
%s, retrying in %s seconds...
Received healthy response to inference request in 2.1644556522369385s
Received healthy response to inference request in 2.1060433387756348s
Received healthy response to inference request in 2.1687605381011963s
Received healthy response to inference request in 2.4547348022460938s
Received healthy response to inference request in 1.992412805557251s
Received healthy response to inference request in 2.158684253692627s
Received healthy response to inference request in 2.220384120941162s
2026-04-07T16:12:39.435340+00:00 monitor updated for chaiml-pony-v3a-g47-lr1_18022_v5
Received healthy response to inference request in 2.160221815109253s
Received healthy response to inference request in 2.1548914909362793s
Received healthy response to inference request in 2.4263057708740234s
Received healthy response to inference request in 2.311628580093384s
Received healthy response to inference request in 2.264482259750366s
Received healthy response to inference request in 2.1126112937927246s
Received healthy response to inference request in 2.101201057434082s
Received healthy response to inference request in 2.2251651287078857s
Received healthy response to inference request in 1.979964017868042s
Received healthy response to inference request in 2.0705747604370117s
Received healthy response to inference request in 2.4247841835021973s
Failed to get response for submission chaiml-mega-v2-top2-g47_10262_v2: HTTPConnectionPool(host='chaiml-mega-v2-top2-g47-10262-v2-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Received healthy response to inference request in 1.9999337196350098s
Received healthy response to inference request in 2.1465630531311035s
Received healthy response to inference request in 2.212489604949951s
Received healthy response to inference request in 2.1467251777648926s
Received healthy response to inference request in 2.0766005516052246s
Received healthy response to inference request in 2.0467681884765625s
Failed to get response for submission chaiml-gspo-glm47-kimi-_50225_v1: ('http://chaiml-gspo-glm47-kimi-50225-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'activator request timeout')
Received healthy response to inference request in 2.07914137840271s
Received healthy response to inference request in 2.056602716445923s
Received healthy response to inference request in 2.782770872116089s
Received healthy response to inference request in 2.459033250808716s
Received healthy response to inference request in 2.2639403343200684s
Received healthy response to inference request in 2.542559862136841s
30 requests
0 failed requests
5th percentile: 1.9957972168922424
10th percentile: 2.042084741592407
20th percentile: 2.075395393371582
30th percentile: 2.104590654373169
40th percentile: 2.146660327911377
50th percentile: 2.15945303440094
60th percentile: 2.186252164840698
70th percentile: 2.2367976903915405
80th percentile: 2.3342597007751467
90th percentile: 2.455164647102356
95th percentile: 2.5049728870391843
99th percentile: 2.713109679222107
mean time: 2.210347819328308
Pipeline stage StressChecker completed in 143.27s
Shutdown handler de-registered
chaiml-pony-v3a-g47-lr1_18022_v5 status is now deployed due to DeploymentManager action
chaiml-pony-v3a-g47-lr1_18022_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v3a-g47-lr1_18022_v5 status is now torndown due to DeploymentManager action