chaiml-pony-v2-q27b-lr1_74562

developer_uid: richhx
submission_id: chaiml-pony-v2-q27b-lr1_74562_v7
model_name: chaiml-pony-v2-q27b-lr1_74562_v7
model_group: ChaiML/pony-v2-q27b-lr1e
status: torndown
timestamp: 2026-03-21T04:31:45+00:00
num_battles: 10827
num_wins: 5659
celo_rating: 1315.56
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/pony-v2-q27b-lr1e4ep1r64g8
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-pony-v2-q27b-lr1_74562_v7
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/pony-v2-q27b-lr1e4ep1r64g8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-17
win_ratio: 0.5226747944952433
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|assistant|>', '<|im_end|>', '<|user|>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-q27b-lr1-74562-v7-uploader
Waiting for job on chaiml-pony-v2-q27b-lr1-74562-v7-uploader to finish
qwen-qwen3-5-27b-v8-uploader: Using quantization_mode: none
qwen-qwen3-5-27b-v8-uploader: Downloading snapshot of Qwen/Qwen3.5-27B...
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Using quantization_mode: none
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Downloading snapshot of ChaiML/pony-v2-q27b-lr1e4ep1r64g8...
qwen-qwen3-5-27b-v8-uploader: Downloaded in 18.562s
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Downloaded in 32.903s
qwen-qwen3-5-27b-v8-uploader: Processed model Qwen/Qwen3.5-27B in 40.606s
qwen-qwen3-5-27b-v8-uploader: creating bucket guanaco-vllm-models
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
qwen-qwen3-5-27b-v8-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   if re.search("-\.", bucket, re.UNICODE):
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   if re.search("\.\.", bucket, re.UNICODE):
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
qwen-qwen3-5-27b-v8-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
qwen-qwen3-5-27b-v8-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
qwen-qwen3-5-27b-v8-uploader: Bucket 's3://guanaco-vllm-models/' created
qwen-qwen3-5-27b-v8-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/.gitattributes
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/tokenizer_config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/chat_template.jinja
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/merges.txt
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/generation_config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/video_preprocessor_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/video_preprocessor_config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors.index.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/preprocessor_config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/README.md
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/vocab.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/tokenizer.json
2026-03-17T23:04:10.750720+00:00 monitor updated for qwen-qwen3-5-27b_v8
2026-03-17T23:04:11.938918+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Processed model ChaiML/pony-v2-q27b-lr1e4ep1r64g8 in 52.978s
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00011-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00011-of-00011.safetensors
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/.gitattributes
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/processor_config.json
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/generation_config.json
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: cp /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/training_args.bin
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/chat_template.jinja
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00005-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00005-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00010-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00010-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00007-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00007-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00009-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00009-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00002-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00002-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00001-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00001-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00006-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00006-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00008-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00008-of-00011.safetensors
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/model-00002-of-00002.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00003-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00003-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00004-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00004-of-00011.safetensors
Job qwen-qwen3-5-27b-v8-uploader completed after 85.71s with status: succeeded
Stopping job with name qwen-qwen3-5-27b-v8-uploader
Pipeline stage VLLMUploader completed in 86.34s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.05s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service qwen-qwen3-5-27b-v8
Waiting for inference service qwen-qwen3-5-27b-v8 to be ready
2026-03-17T23:05:10.889515+00:00 monitor updated for qwen-qwen3-5-27b_v8
2026-03-17T23:05:12.086096+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
2026-03-17T23:06:11.036564+00:00 monitor updated for qwen-qwen3-5-27b_v8
2026-03-17T23:06:12.230745+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
Waiting for job on qwen-qwen3-5-27b-v8-uploader to finish
chaiml-pony-v2-q27b-lr1_74562_v7 status is now pending due to richhx requesting redeployment of chaiml-pony-v2-q27b-lr1_74562_v7
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-pony-v2-q27b-lr1-74562-v7-uploader
Job failed chaiml-pony-v2-q27b-lr1-74562-v7-uploader: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Tue, 17 Mar 2026 23:24:32 GMT', 'Content-Type': 'application/json', 'Content-Length': '282', 'Connection': 'keep-alive', 'audit-id': 'fa1bbe4f-dfed-49ba-9b49-6272540c437c', 'Cache-Control': 'no-cache, private', 'x-kubernetes-pf-flowschema-uid': '757159af-7462-4610-a5a7-751d984daa90', 'x-kubernetes-pf-prioritylevel-uid': '5f6bf145-f133-455b-a330-c9bf3e5ea771', 'cf-cache-status': 'DYNAMIC', 'set-cookie': '__cf_bm=_SNAAS3WEM0jX2CP6y._._.QMvprAlf15eHhMmNNan8-1773789872.034718-1.0.1.1-hc6HaPmCkxA24xkqnQyCHqEcBV9joInm_KbUNBUNpY1FGgVgR1xzDtxRBCS_7v.eNG8FVV_C2Ch0UYOJ4ee7Z; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Tue, 17 Mar 2026 23:54:32 GMT', 'Server': 'cloudflare', 'CF-RAY': '9ddfc06c3d5fc7c1-IAD'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"chaiml-pony-v2-q27b-lr1-74562-v7-uploader\" already exists","reason":"AlreadyExists","details":{"name":"chaiml-pony-v2-q27b-lr1-74562-v7-uploader","group":"batch","kind":"jobs"},"code":409}

Stopping job with name chaiml-pony-v2-q27b-lr1-74562-v7-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-pony-v2-q27b-lr1-74562-v7-uploader
Waiting for job on chaiml-pony-v2-q27b-lr1-74562-v7-uploader to finish
qwen-qwen3-5-27b-v8-uploader: Using quantization_mode: none
qwen-qwen3-5-27b-v8-uploader: Downloading snapshot of Qwen/Qwen3.5-27B...
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Using quantization_mode: none
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Downloading snapshot of ChaiML/pony-v2-q27b-lr1e4ep1r64g8...
qwen-qwen3-5-27b-v8-uploader: Downloaded in 18.336s
qwen-qwen3-5-27b-v8-uploader: Processed model Qwen/Qwen3.5-27B in 39.171s
qwen-qwen3-5-27b-v8-uploader: creating bucket guanaco-vllm-models
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
qwen-qwen3-5-27b-v8-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   if re.search("-\.", bucket, re.UNICODE):
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Downloaded in 35.005s
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
qwen-qwen3-5-27b-v8-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
qwen-qwen3-5-27b-v8-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
qwen-qwen3-5-27b-v8-uploader: Bucket 's3://guanaco-vllm-models/' created
qwen-qwen3-5-27b-v8-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/.gitattributes": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/README.md s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/README.md": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/chat_template.jinja": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/config.json": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/generation_config.json": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/merges.txt": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00001-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00001-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00002-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00002-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00003-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00003-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00004-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00004-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00005-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00005-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00006-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00006-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00007-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00007-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00008-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00008-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00009-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00009-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00010-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00010-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors-00011-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00011-of-00011.safetensors": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors.index.json": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/preprocessor_config.json": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/tokenizer.json": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/tokenizer_config.json": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/video_preprocessor_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/video_preprocessor_config.json": object size matches
qwen-qwen3-5-27b-v8-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/vocab.json": object size matches
2026-03-17T23:25:30.619120+00:00 monitor updated for qwen-qwen3-5-27b_v8
2026-03-17T23:25:31.375740+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
Job qwen-qwen3-5-27b-v8-uploader completed after 66.6s with status: succeeded
Stopping job with name qwen-qwen3-5-27b-v8-uploader
Pipeline stage VLLMUploader completed in 67.71s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.67s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Pipeline stage VLLMDeployer completed in 0.52s
run pipeline stage %s
Running pipeline stage StressChecker
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Processed model ChaiML/pony-v2-q27b-lr1e4ep1r64g8 in 56.264s
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: creating bucket guanaco-vllm-models
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/.gitattributes": object size matches
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/README.md": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/added_tokens.json": object size matches
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/args.json": object size matches
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/chat_template.jinja": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/config.json": object size matches
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/generation_config.json": object size matches
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/merges.txt": object size matches
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/model-00001-of-00002.safetensors": object size matches
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/model-00002-of-00002.safetensors": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/model.safetensors.index.json": object size matches
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/preprocessor_config.json": object size matches
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/processor_config.json": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/special_tokens_map.json": object size matches
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/tokenizer.json": object size matches
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/tokenizer_config.json": object size matches
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/trainer_state.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/trainer_state.json": object size matches
Received unhealthy response to inference request!
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/training_args.bin": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/vocab.json": object size matches
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Job chaiml-pony-v2-q27b-lr1-74562-v7-uploader completed after 78.95s with status: succeeded
Received unhealthy response to inference request!
Stopping job with name chaiml-pony-v2-q27b-lr1-74562-v7-uploader
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Pipeline stage VLLMUploader completed in 80.48s
Received unhealthy response to inference request!
run pipeline stage %s
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Running pipeline stage VLLMTemplater
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
Pipeline stage VLLMTemplater completed in 0.61s
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
run pipeline stage %s
Received unhealthy response to inference request!
Running pipeline stage VLLMDeployer
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Creating inference service chaiml-pony-v2-q27b-lr1-74562-v7
Received unhealthy response to inference request!
Waiting for inference service chaiml-pony-v2-q27b-lr1-74562-v7 to be ready
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
30 requests
30 failed requests
5th percentile: 0.3620994210243225
10th percentile: 0.388038969039917
20th percentile: 0.418794584274292
30th percentile: 0.44139201641082765
40th percentile: 0.45626029968261717
50th percentile: 0.4753676652908325
60th percentile: 0.489064884185791
70th percentile: 0.5208352088928222
80th percentile: 0.5419211864471436
90th percentile: 0.5626179218292237
95th percentile: 0.5809054493904113
99th percentile: 1.286787848472596
mean time: 0.5082311630249023
%s, retrying in %s seconds...
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
30 requests
30 failed requests
5th percentile: 0.420101010799408
10th percentile: 0.42135984897613527
20th percentile: 0.4275675773620605
30th percentile: 0.432767653465271
40th percentile: 0.4340821743011475
50th percentile: 0.43669676780700684
60th percentile: 0.43839011192321775
70th percentile: 0.44327282905578613
80th percentile: 0.44802107810974123
90th percentile: 0.46075048446655276
95th percentile: 0.488299524784088
99th percentile: 0.6541293954849245
mean time: 0.44797252813975014
%s, retrying in %s seconds...
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
{"detail":"('http://qwen-qwen3-5-27b-v8-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '{\"error\":{\"message\":\"The model `Qwen/Qwen3.5-27B` does not exist.\",\"type\":\"NotFoundError\",\"param\":\"model\",\"code\":404}}')"}
Received unhealthy response to inference request!
30 requests
30 failed requests
5th percentile: 0.4160201191902161
10th percentile: 0.41882710456848143
20th percentile: 0.42454776763916013
30th percentile: 0.43578238487243653
40th percentile: 0.43802390098571775
50th percentile: 0.4435386657714844
60th percentile: 0.45051941871643064
70th percentile: 0.4573859453201294
80th percentile: 0.46017985343933104
90th percentile: 0.4624758243560791
95th percentile: 0.4710869669914245
2026-03-17T23:26:30.813469+00:00 monitor updated for qwen-qwen3-5-27b_v8
99th percentile: 0.6747014284133913
mean time: 0.45323173999786376
2026-03-17T23:26:31.566605+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service qwen-qwen3-5-27b-v8 is running
Tearing down inference service qwen-qwen3-5-27b-v8
Service qwen-qwen3-5-27b-v8 has been torndown
Pipeline stage VLLMDeleter completed in 1.15s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key qwen-qwen3-5-27b-v8/default/.gitattributes from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/README.md from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/config.json from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/generation_config.json from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/merges.txt from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00001-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00002-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00003-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00004-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00005-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00006-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00007-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00008-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00009-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00010-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors-00011-of-00011.safetensors from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/preprocessor_config.json from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/tokenizer_config.json from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/video_preprocessor_config.json from bucket guanaco-vllm-models
Deleting key qwen-qwen3-5-27b-v8/default/vocab.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 11.75s
Shutdown handler de-registered
DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
qwen-qwen3-5-27b_v8 status is now failed due to DeploymentManager action
2026-03-17T23:27:31.820682+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
2026-03-17T23:28:31.988104+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
qwen-qwen3-5-27b_v8 status is now pending due to richhx requesting redeployment of qwen-qwen3-5-27b_v8
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
chaiml-pony-v2-q27b-lr1_74562_v7 status is now pending due to richhx requesting redeployment of chaiml-pony-v2-q27b-lr1_74562_v7
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage VLLMUploader
run pipeline %s
Starting job with name qwen-qwen3-5-27b-v8-uploader
run pipeline stage %s
Running pipeline stage VLLMUploader
Waiting for job on qwen-qwen3-5-27b-v8-uploader to finish
Starting job with name chaiml-pony-v2-q27b-lr1-74562-v7-uploader
Waiting for job on chaiml-pony-v2-q27b-lr1-74562-v7-uploader to finish
qwen-qwen3-5-27b-v8-uploader: Using quantization_mode: none
qwen-qwen3-5-27b-v8-uploader: Downloading snapshot of Qwen/Qwen3.5-27B...
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Using quantization_mode: none
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Downloading snapshot of ChaiML/pony-v2-q27b-lr1e4ep1r64g8...
qwen-qwen3-5-27b-v8-uploader: Downloaded in 18.318s
2026-03-17T23:29:32.157188+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Downloaded in 33.665s
qwen-qwen3-5-27b-v8-uploader: Processed model Qwen/Qwen3.5-27B in 40.504s
qwen-qwen3-5-27b-v8-uploader: creating bucket guanaco-vllm-models
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
qwen-qwen3-5-27b-v8-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   if re.search("-\.", bucket, re.UNICODE):
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
qwen-qwen3-5-27b-v8-uploader:   if re.search("\.\.", bucket, re.UNICODE):
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
qwen-qwen3-5-27b-v8-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
qwen-qwen3-5-27b-v8-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
qwen-qwen3-5-27b-v8-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
qwen-qwen3-5-27b-v8-uploader: Bucket 's3://guanaco-vllm-models/' created
qwen-qwen3-5-27b-v8-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/.gitattributes
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors.index.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/README.md
2026-03-17T23:29:56.290130+00:00 monitor updated for qwen-qwen3-5-27b_v8
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/preprocessor_config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/generation_config.json
2026-03-17T23:29:57.029254+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/chat_template.jinja
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/tokenizer_config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/merges.txt
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/video_preprocessor_config.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/video_preprocessor_config.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/tokenizer.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/vocab.json
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00010-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00010-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00001-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00001-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00008-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00008-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00007-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00007-of-00011.safetensors
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00003-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00003-of-00011.safetensors
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Processed model ChaiML/pony-v2-q27b-lr1e4ep1r64g8 in 54.700s
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: creating bucket guanaco-vllm-models
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-pony-v2-q27b-lr1-74562-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/.gitattributes": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/README.md": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/added_tokens.json": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/args.json": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/chat_template.jinja": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/config.json": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/generation_config.json": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/merges.txt": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/model-00001-of-00002.safetensors": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/model-00002-of-00002.safetensors": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/model.safetensors.index.json": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/preprocessor_config.json": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/processor_config.json": object size matches
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/special_tokens_map.json": object size matches
qwen-qwen3-5-27b-v8-uploader: cp /dev/shm/model_output/model.safetensors-00006-of-00011.safetensors s3://guanaco-vllm-models/qwen-qwen3-5-27b-v8/default/model.safetensors-00006-of-00011.safetensors
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/tokenizer.json": object size matches
Job qwen-qwen3-5-27b-v8-uploader completed after 82.59s with status: succeeded
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/tokenizer_config.json": object size matches
Stopping job with name qwen-qwen3-5-27b-v8-uploader
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/trainer_state.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/trainer_state.json": object size matches
Pipeline stage VLLMUploader completed in 84.51s
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/training_args.bin": object size matches
run pipeline stage %s
chaiml-pony-v2-q27b-lr1-74562-v7-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-pony-v2-q27b-lr1-74562-v7/default/vocab.json": object size matches
Running pipeline stage VLLMTemplater
Job chaiml-pony-v2-q27b-lr1-74562-v7-uploader completed after 83.48s with status: succeeded
Stopping job with name chaiml-pony-v2-q27b-lr1-74562-v7-uploader
Pipeline stage VLLMTemplater completed in 0.66s
Pipeline stage VLLMUploader completed in 85.18s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Running pipeline stage VLLMTemplater
Creating inference service qwen-qwen3-5-27b-v8
Pipeline stage VLLMTemplater completed in 0.60s
Waiting for inference service qwen-qwen3-5-27b-v8 to be ready
run pipeline stage %s
Running pipeline stage VLLMDeployer
Pipeline stage VLLMDeployer completed in 0.53s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-17T23:30:32.497637+00:00 monitor updated for chaiml-pony-v2-q27b-lr1_74562_v7
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
Received unhealthy response to inference request!
{"detail":"('http://chaiml-pony-v2-q27b-lr1-74562-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', '')"}
chaiml-pony-v2-q27b-lr1_74562_v7 status is now inactive due to auto deactivation removed underperforming models
chaiml-pony-v2-q27b-lr1_74562_v7 status is now torndown due to DeploymentManager action