chaiml-pony-v2-q27b-lr1_74562

submission_id: chaiml-pony-v2-q27b-lr1_74562_v4
developer_uid: zonemercy
status: torndown
model_repo: ChaiML/pony-v2-q27b-lr1e4ep1r64g8
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
timestamp: 2026-03-17T14:39:28+00:00
model_name: chaiml-pony-v2-q27b-lr1_74562_v4
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-q27b-lr1-74562-v4-uploader
Waiting for job on chaiml-pony-v2-q27b-lr1-74562-v4-uploader to finish
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: Using quantization_mode: none
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: Downloading snapshot of ChaiML/pony-v2-q27b-lr1e4ep1r64g8...
qwen-qwen3-5-27b-v6-uploader: Downloaded in 18.756s
qwen-qwen3-5-27b-v6-uploader: Processed model Qwen/Qwen3.5-27B in 39.423s
qwen-qwen3-5-27b-v6-uploader: creating bucket guanaco-vllm-models
qwen-qwen3-5-27b-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v6-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
qwen-qwen3-5-27b-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
qwen-qwen3-5-27b-v6-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
qwen-qwen3-5-27b-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v6-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
qwen-qwen3-5-27b-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v6-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
qwen-qwen3-5-27b-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: Downloaded in 33.289s
qwen-qwen3-5-27b-v6-uploader:   if re.search("-\.", bucket, re.UNICODE):
qwen-qwen3-5-27b-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v6-uploader:   if re.search("\.\.", bucket, re.UNICODE):
qwen-qwen3-5-27b-v6-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
qwen-qwen3-5-27b-v6-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
qwen-qwen3-5-27b-v6-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
qwen-qwen3-5-27b-v6-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
qwen-qwen3-5-27b-v6-uploader: Bucket 's3://guanaco-vllm-models/' created
qwen-qwen3-5-27b-v6-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/chat_template.jinja
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/config.json
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors.index.json
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/README.md
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/.gitattributes
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/video_preprocessor_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/video_preprocessor_config.json
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/tokenizer_config.json
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/preprocessor_config.json
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/generation_config.json
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/merges.txt
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/vocab.json
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/tokenizer.json
2026-03-17T10:11:10.479999+00:00 monitor updated for qwen-qwen3-5-27b_v6
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00005-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00005-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00010-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00010-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00009-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00009-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00007-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00007-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00006-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00006-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00002-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00002-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00003-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00003-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00004-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00004-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00008-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00008-of-00011.safetensors
qwen-qwen3-5-27b-v6-uploader: cp /dev/shm/model_output/model.safetensors-00001-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v6/default/model.safetensors-00001-of-00011.safetensors
2026-03-17T10:11:21.592892+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v4
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: Processed model ChaiML/pony-v2-q27b-lr1e4ep1r64g8 in 53.230s
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-q27b-lr1-74562-v4-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v4/default
Job qwen-qwen3-5-27b-v6-uploader completed after 76.3s with status: succeeded
Stopping job with name qwen-qwen3-5-27b-v6-uploader
Pipeline stage VLLMUploader completed in 78.68s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.76s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service qwen-qwen3-5-27b-v6
Waiting for inference service qwen-qwen3-5-27b-v6 to be ready
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v4/default/model-00002-of-00002.safetensors
2026-03-17T10:12:10.662252+00:00 monitor updated for qwen-qwen3-5-27b_v6
2026-03-17T10:12:21.759693+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v4
2026-03-17T10:13:10.835447+00:00 monitor updated for qwen-qwen3-5-27b_v6
2026-03-17T10:13:21.995040+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v4
chaiml-pony-v2-q27b-lr1-74562-v4-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v4/default/model-00001-of-00002.safetensors
2026-03-17T10:14:11.017404+00:00 monitor updated for qwen-qwen3-5-27b_v6
Job chaiml-pony-v2-q27b-lr1-74562-v4-uploader completed after 236.45s with status: succeeded
Stopping job with name chaiml-pony-v2-q27b-lr1-74562-v4-uploader
Pipeline stage VLLMUploader completed in 237.40s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.56s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-pony-v2-q27b-lr1-74562-v4
Waiting for inference service chaiml-pony-v2-q27b-lr1-74562-v4 to be ready
2026-03-17T10:14:22.271333+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v4
Inference service qwen-qwen3-5-27b-v6 ready after 180.60272431373596s
Pipeline stage VLLMDeployer completed in 181.50s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-17T10:15:11.195852+00:00 monitor updated for qwen-qwen3-5-27b_v6
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-17T10:15:22.457500+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v4
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1_74562_v4 status is now failed due to DeploymentManager action
chaiml-pony-v2-q27b-lr1_74562_v4 status is now torndown due to DeploymentManager action
chaiml-pony-v2-q27b-lr1_74562_v4 status is now torndown due to DeploymentManager action