chaiml-prm-kimi-v1-300_92220

submission_id: chaiml-prm-kimi-v1-300_92220_v12
developer_uid: zonemercy
status: torndown
model_repo: ChaiML/prm_kimi_v1_300k_default8b-cosine-lr1e6g32
generation_params: {'temperature': 0.5, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': 'User: {message}\n', 'response_template': '', 'truncate_by_message': True}
timestamp: 2026-03-27T16:56:16+00:00
model_name: training123
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-prm-kimi-v1-300-92220-v12-uploader
Waiting for job on chaiml-prm-kimi-v1-300-92220-v12-uploader to finish
chaiml-prm-kimi-v1-300-92220-v12-uploader: Using quantization_mode: none
chaiml-prm-kimi-v1-300-92220-v12-uploader: Downloading snapshot of ChaiML/prm_kimi_v1_300k_default8b-cosine-lr1e6g32...
chaiml-prm-kimi-v1-300-92220-v12-uploader: Downloaded in 14.272s
chaiml-prm-kimi-v1-300-92220-v12-uploader: Processed model ChaiML/prm_kimi_v1_300k_default8b-cosine-lr1e6g32 in 19.862s
chaiml-prm-kimi-v1-300-92220-v12-uploader: creating bucket guanaco-vllm-models
chaiml-prm-kimi-v1-300-92220-v12-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v12-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-prm-kimi-v1-300-92220-v12-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-prm-kimi-v1-300-92220-v12-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-prm-kimi-v1-300-92220-v12-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v12-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-prm-kimi-v1-300-92220-v12-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v12-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-prm-kimi-v1-300-92220-v12-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v12-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-prm-kimi-v1-300-92220-v12-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v12-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-prm-kimi-v1-300-92220-v12-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-prm-kimi-v1-300-92220-v12-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-prm-kimi-v1-300-92220-v12-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-prm-kimi-v1-300-92220-v12-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-prm-kimi-v1-300-92220-v12-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-prm-kimi-v1-300-92220-v12-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/special_tokens_map.json
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/config.json
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/README.md
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/.gitattributes
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/model.safetensors.index.json
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/tokenizer_config.json
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/tokenizer.json
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/model-00004-of-00004.safetensors
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/model-00003-of-00004.safetensors
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/model-00001-of-00004.safetensors
chaiml-prm-kimi-v1-300-92220-v12-uploader: cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v12/default/model-00002-of-00004.safetensors
Job chaiml-prm-kimi-v1-300-92220-v12-uploader completed after 43.39s with status: succeeded
Stopping job with name chaiml-prm-kimi-v1-300-92220-v12-uploader
Pipeline stage VLLMUploader completed in 43.89s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.37s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-prm-kimi-v1-300-92220-v12
Waiting for inference service chaiml-prm-kimi-v1-300-92220-v12 to be ready
2026-03-27T16:29:21.958726+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:30:22.088314+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:31:22.182333+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:32:22.287031+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:33:22.375364+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:34:22.461300+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:35:22.557229+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:36:22.672323+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:37:22.792383+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:38:22.918512+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:39:23.009536+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:40:23.102175+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:41:34.735373+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:42:34.830543+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:43:34.925257+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:44:35.039803+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:45:35.127291+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:46:35.220949+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:47:35.316045+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:48:35.420347+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:49:35.520593+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:50:35.645901+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
2026-03-27T16:51:35.775505+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v12
Tearing down inference service chaiml-prm-kimi-v1-300-92220-v12
clean up pipeline due to error=DeploymentError('404\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({\'Date\': \'Fri, 27 Mar 2026 16:51:42 GMT\', \'Content-Type\': \'application/json\', \'Content-Length\': \'304\', \'Connection\': \'keep-alive\', \'audit-id\': \'5f1df0dc-fd67-4278-8952-d691e5fbb8f4\', \'Cache-Control\': \'no-cache, private\', \'x-kubernetes-pf-flowschema-uid\': \'757159af-7462-4610-a5a7-751d984daa90\', \'x-kubernetes-pf-prioritylevel-uid\': \'5f6bf145-f133-455b-a330-c9bf3e5ea771\', \'cf-cache-status\': \'DYNAMIC\', \'set-cookie\': \'__cf_bm=ZxdosbYNh_GM04Qwqr2HXkv1w1HBd9wGRP75qGM70NI-1774630302.7436316-1.0.1.1-Hs.7ig8874tyI1XngddNKBHMnaX5h0tEH3LFCr2u9Zfo.0MYzgHKm9s0wAonbA_Ge7.zk2LJWCek4rojY7gUyfpNx3.YYnJCiT6fBYgYlHvu20CChxq5O4zpoHWqj_XH; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Fri, 27 Mar 2026 17:21:42 GMT\', \'Server\': \'cloudflare\', \'CF-RAY\': \'9e2fe6c02c013e80-IAD\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io \\\\"chaiml-prm-kimi-v1-300-92220-v12\\\\" not found","reason":"NotFound","details":{"name":"chaiml-prm-kimi-v1-300-92220-v12","group":"serving.kserve.io","kind":"inferenceservices"},"code":404}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 415, in request\n    return self.rest_client.DELETE(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/rest.py", line 270, in DELETE\n    return self.request("DELETE", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-prm-kimi-v1-300-92220-v12 is running
Skipping teardown as no inference service was found
Pipeline stage VLLMDeleter completed in 0.46s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Pipeline stage VLLMModelDeleter completed in 0.32s
Shutdown handler de-registered
chaiml-prm-kimi-v1-300_92220_v12 status is now failed due to DeploymentManager action
chaiml-prm-kimi-v1-300_92220_v12 status is now torndown due to DeploymentManager action