evelyn777-chai-sft-3b-v5

submission_id: evelyn777-chai-sft-3b-v5_v2
developer_uid: chai_evaluation_service
status: failed
model_repo: evelyn777/chai-sft-3b-v5
generation_params: {'temperature': 0.7, 'top_p': 0.9, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
timestamp: 2026-02-08T08:44:39+00:00
model_name: evelyn777-chai-sft-3b-v5_v2
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name evelyn777-chai-sft-3b-v5-v2-uploader
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Waiting for job on evelyn777-chai-sft-3b-v5-v2-uploader to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
evelyn777-chai-sft-3b-v5-v2-uploader: Using quantization_mode: none
evelyn777-chai-sft-3b-v5-v2-uploader: Downloading snapshot of evelyn777/chai-sft-3b-v5...
evelyn777-chai-sft-3b-v5-v2-uploader: 
Fetching 13 files:   0%|          | 0/13 [00:00<?, ?it/s]
Fetching 13 files:   8%|▊         | 1/13 [00:00<00:03,  3.19it/s]
Fetching 13 files:  54%|█████▍    | 7/13 [00:04<00:03,  1.56it/s]
Fetching 13 files: 100%|██████████| 13/13 [00:04<00:00,  2.97it/s]
evelyn777-chai-sft-3b-v5-v2-uploader: Downloaded in 4.494s
evelyn777-chai-sft-3b-v5-v2-uploader: Processed model evelyn777/chai-sft-3b-v5 in 6.977s
evelyn777-chai-sft-3b-v5-v2-uploader: creating bucket guanaco-vllm-models
evelyn777-chai-sft-3b-v5-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
evelyn777-chai-sft-3b-v5-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
evelyn777-chai-sft-3b-v5-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
evelyn777-chai-sft-3b-v5-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
evelyn777-chai-sft-3b-v5-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
evelyn777-chai-sft-3b-v5-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
evelyn777-chai-sft-3b-v5-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
evelyn777-chai-sft-3b-v5-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
evelyn777-chai-sft-3b-v5-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
evelyn777-chai-sft-3b-v5-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
evelyn777-chai-sft-3b-v5-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
evelyn777-chai-sft-3b-v5-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
evelyn777-chai-sft-3b-v5-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
evelyn777-chai-sft-3b-v5-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
evelyn777-chai-sft-3b-v5-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
evelyn777-chai-sft-3b-v5-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
evelyn777-chai-sft-3b-v5-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
evelyn777-chai-sft-3b-v5-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/.gitattributes
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/config.json
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/special_tokens_map.json
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/generation_config.json
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/tokenizer_config.json
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/chat_template.jinja
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/model.safetensors.index.json
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/added_tokens.json
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/merges.txt
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/vocab.json
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/tokenizer.json
HTTP Request: %s %s "%s %d %s"
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/model-00002-of-00002.safetensors
evelyn777-chai-sft-3b-v5-v2-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/evelyn777-chai-sft-3b-v5-v2/model-00001-of-00002.safetensors
Job evelyn777-chai-sft-3b-v5-v2-uploader completed after 82.91s with status: succeeded
Stopping job with name evelyn777-chai-sft-3b-v5-v2-uploader
Pipeline stage VLLMUploader completed in 85.17s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service evelyn777-chai-sft-3b-v5-v2
Waiting for inference service evelyn777-chai-sft-3b-v5-v2 to be ready
Failed to get response for submission chaiml-muster-v0a-lr1e5_44160_v5: HTTPConnectionPool(host='chaiml-muster-v0a-lr1e5-44160-v5-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Tearing down inference service evelyn777-chai-sft-3b-v5-v2
clean up pipeline due to error=DeploymentError("Exception when calling CustomObjectsApi->get_namespaced_custom_object:                        (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Date': 'Sun, 08 Feb 2026 08:44:37 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'CF-RAY': '9ca9d89b8aeb2899-IAD', 'cf-cache-status': 'DYNAMIC', 'Set-Cookie': '__cf_bm=-1770540277-1.0.1.1-A7rttCZJX3oI_zGSERMlDIN8WkqPUzGl2bZPf93tjpig.ueY0umUsvQtLsYtXM15nZTUmhGAGXUfWA.BVEFNjRj9Mw6YX__uM_0BekML_EI; path=/; expires=Sun, 08-Feb-26 09:14:37 GMT; domain=.coreweave.com; HttpOnly; Secure; SameSite=None', 'Server': 'cloudflare'})\n\n")
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.19s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key evelyn777-chai-sft-3b-v5-v2/.gitattributes from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/added_tokens.json from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/chat_template.jinja from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/config.json from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/generation_config.json from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/merges.txt from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/model-00001-of-00002.safetensors from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/model-00002-of-00002.safetensors from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/tokenizer.json from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/tokenizer_config.json from bucket guanaco-vllm-models
Deleting key evelyn777-chai-sft-3b-v5-v2/vocab.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 1.99s
Shutdown handler de-registered
evelyn777-chai-sft-3b-v5_v2 status is now failed due to DeploymentManager action