chaiml-prm-kimi-v1-300_92220

developer_uid: zonemercy

submission_id: chaiml-prm-kimi-v1-300_92220_v13

model_name: training123

model_group: ChaiML/prm_kimi_v1_300k_

status: torndown

timestamp: 2026-03-31T06:21:42+00:00

num_battles: 10800

num_wins: 5768

celo_rating: 1316.7

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/prm_kimi_v1_300k_default8b-cosine-lr1e6g32

model_architecture: LlamaForSequenceClassification

model_num_parameters: 8030261248.0

best_of: 8

max_input_tokens: 2048

max_output_tokens: 1

reward_model: default

display_name: training123

ineligible_reason: max_output_tokens!=64

is_internal_developer: True

language_model: ChaiML/prm_kimi_v1_300k_default8b-cosine-lr1e6g32

model_size: 8B

ranking_group: single

us_pacific_date: 2026-03-27

win_ratio: 0.534074074074074

generation_params: {'temperature': 0.5, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '\n'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 1}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': 'User: {message}\n', 'response_template': '', 'truncate_by_message': True}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-prm-kimi-v1-300-92220-v13-uploader
Waiting for job on chaiml-prm-kimi-v1-300-92220-v13-uploader to finish
chaiml-prm-kimi-v1-300-92220-v13-uploader: Using quantization_mode: none
chaiml-prm-kimi-v1-300-92220-v13-uploader: Downloading snapshot of ChaiML/prm_kimi_v1_300k_default8b-cosine-lr1e6g32...
chaiml-prm-kimi-v1-300-92220-v13-uploader: Processed model ChaiML/prm_kimi_v1_300k_default8b-cosine-lr1e6g32 in 17.951s
chaiml-prm-kimi-v1-300-92220-v13-uploader: creating bucket guanaco-vllm-models
chaiml-prm-kimi-v1-300-92220-v13-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v13-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-prm-kimi-v1-300-92220-v13-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-prm-kimi-v1-300-92220-v13-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-prm-kimi-v1-300-92220-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v13-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-prm-kimi-v1-300-92220-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v13-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-prm-kimi-v1-300-92220-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v13-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-prm-kimi-v1-300-92220-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-prm-kimi-v1-300-92220-v13-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-prm-kimi-v1-300-92220-v13-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-prm-kimi-v1-300-92220-v13-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-prm-kimi-v1-300-92220-v13-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-prm-kimi-v1-300-92220-v13-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-prm-kimi-v1-300-92220-v13-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-prm-kimi-v1-300-92220-v13-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/.gitattributes
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/README.md
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/config.json
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/tokenizer_config.json
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/special_tokens_map.json
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/model.safetensors.index.json
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/tokenizer.json
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/model-00004-of-00004.safetensors s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/model-00004-of-00004.safetensors
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/model-00003-of-00004.safetensors s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/model-00003-of-00004.safetensors
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/model-00001-of-00004.safetensors s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/model-00001-of-00004.safetensors
chaiml-prm-kimi-v1-300-92220-v13-uploader: cp /dev/shm/model_output/model-00002-of-00004.safetensors s3://guanaco-vllm-models/chaiml-prm-kimi-v1-300-92220-v13/default/model-00002-of-00004.safetensors
Job chaiml-prm-kimi-v1-300-92220-v13-uploader completed after 50.26s with status: succeeded
Stopping job with name chaiml-prm-kimi-v1-300-92220-v13-uploader
Pipeline stage VLLMUploader completed in 50.97s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.85s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-prm-kimi-v1-300-92220-v13
Waiting for inference service chaiml-prm-kimi-v1-300-92220-v13 to be ready
2026-03-28T02:52:31.895516+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T02:53:31.979905+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T02:54:32.063779+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T02:55:32.152788+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T02:56:32.247551+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T02:57:32.379309+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T02:58:32.551443+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T02:59:32.639657+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:00:32.730202+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:01:32.820454+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:02:32.914245+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:03:32.999795+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:04:33.088028+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:05:33.173361+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:06:33.257202+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:07:33.346731+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:08:33.475155+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:09:33.564239+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:10:33.655081+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
2026-03-28T03:11:33.738995+00:00 monitor updated for chaiml-prm-kimi-v1-300_92220_v13
Tearing down inference service chaiml-prm-kimi-v1-300-92220-v13
clean up pipeline due to error=DeploymentError('404\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({\'Date\': \'Sat, 28 Mar 2026 03:12:24 GMT\', \'Content-Type\': \'application/json\', \'Content-Length\': \'304\', \'Connection\': \'keep-alive\', \'audit-id\': \'7c171a0b-a066-4a9e-85ed-4c6869bdc0c0\', \'Cache-Control\': \'no-cache, private\', \'x-kubernetes-pf-flowschema-uid\': \'757159af-7462-4610-a5a7-751d984daa90\', \'x-kubernetes-pf-prioritylevel-uid\': \'5f6bf145-f133-455b-a330-c9bf3e5ea771\', \'cf-cache-status\': \'DYNAMIC\', \'set-cookie\': \'__cf_bm=TGRxYpLGpDtzXTT4iLag7SEihc6mobx.H2XUvv3qIvY-1774667544.6148052-1.0.1.1-FosvYWa1ZW9Mws1NJx77k_G_Fv4nYYvZCuBGyEdjQ_ZFHGCHVTSnAsifQU9_HCeNrXEPbujdXD7udWP7wFrAebK6hQk9KOYR_ELB0.vl4FG6fPaDOLGqfNSZp9QqobvN; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Sat, 28 Mar 2026 03:42:24 GMT\', \'Server\': \'cloudflare\', \'CF-RAY\': \'9e3373f9dda59c48-IAD\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io \\\\"chaiml-prm-kimi-v1-300-92220-v13\\\\" not found","reason":"NotFound","details":{"name":"chaiml-prm-kimi-v1-300-92220-v13","group":"serving.kserve.io","kind":"inferenceservices"},"code":404}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 415, in request\n    return self.rest_client.DELETE(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/rest.py", line 270, in DELETE\n    return self.request("DELETE", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.12/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-prm-kimi-v1-300-92220-v13 is running
Skipping teardown as no inference service was found
Pipeline stage VLLMDeleter completed in 0.36s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Pipeline stage VLLMModelDeleter completed in 0.36s
Shutdown handler de-registered
chaiml-prm-kimi-v1-300_92220_v13 status is now failed due to DeploymentManager action
chaiml-prm-kimi-v1-300_92220_v13 status is now torndown due to DeploymentManager action
chaiml-prm-kimi-v1-300_92220_v13 status is now deployed due to DeploymentManager action
chaiml-prm-kimi-v1-300_92220_v13 status is now inactive due to auto deactivation removed underperforming models
chaiml-prm-kimi-v1-300_92220_v13 status is now torndown due to DeploymentManager action