chaiml-gspo-glm47-combi_63111

submission_id: chaiml-gspo-glm47-combi_63111_v2

developer_uid: chai_backend_admin

status: torndown

model_repo: ChaiML/gspo-glm47-combine-rm82-zy-data-step200

generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|user|>', '<|assistant|>', '</s>', '<|im_end|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}

formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}

timestamp: 2026-03-18T19:50:59+00:00

model_name: chaiml-gspo-glm47-combi_63111_v2

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-63111-v2-uploader
Waiting for job on chaiml-gspo-glm47-combi-63111-v2-uploader to finish
chaiml-gspo-glm47-combi-63111-v2-uploader: Using quantization_mode: fp8
chaiml-gspo-glm47-combi-63111-v2-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-zy-data-step200-FP8 already exists in ChaiML
chaiml-gspo-glm47-combi-63111-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-gspo-glm47-combi-63111-v2-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm82-zy-data-step200-FP8...
chaiml-gspo-glm47-combi-63111-v2-uploader: Downloaded in 0.176s
chaiml-gspo-glm47-combi-63111-v2-uploader: Processed model ChaiML/gspo-glm47-combine-rm82-zy-data-step200 in 3.688s
chaiml-gspo-glm47-combi-63111-v2-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-63111-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-63111-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-63111-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-63111-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-63111-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-63111-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-63111-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-63111-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-63111-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-63111-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-63111-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-63111-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-63111-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-63111-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v2/default
chaiml-gspo-glm47-combi-63111-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-63111-v2/default/.gitattributes
Job chaiml-gspo-glm47-combi-63111-v2-uploader completed after 51.94s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-63111-v2-uploader
Pipeline stage VLLMUploader completed in 52.41s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.52s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-63111-v2
Waiting for inference service chaiml-gspo-glm47-combi-63111-v2 to be ready
2026-03-18T19:41:21.327871+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v2
2026-03-18T19:42:21.425777+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v2
2026-03-18T19:43:21.517579+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v2
2026-03-18T19:44:21.601728+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v2
2026-03-18T19:45:21.696229+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v2
2026-03-18T19:46:21.792367+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v2
2026-03-18T19:47:21.894205+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v2
2026-03-18T19:48:22.007348+00:00 monitor updated for chaiml-gspo-glm47-combi_63111_v2
Tearing down inference service chaiml-gspo-glm47-combi-63111-v2
clean up pipeline due to error=DeploymentError('404\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'3569ff6a-e55f-4b4a-afce-6f258ff9b789, 71fd9e65-5e89-4684-882f-085131bce83d\', \'Cache-Control\': \'no-cache, private, no-cache, private\', \'Content-Length\': \'304\', \'Content-Type\': \'application/json\', \'Date\': \'Wed, 18 Mar 2026 19:49:06 GMT\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'6d61b0c7-bf04-4f40-8d77-09fab6d530e3\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'575fa237-17f0-45cc-909e-dda8024470d8\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io \\\\"chaiml-gspo-glm47-combi-63111-v2\\\\" not found","reason":"NotFound","details":{"name":"chaiml-gspo-glm47-combi-63111-v2","group":"serving.kserve.io","kind":"inferenceservices"},"code":404}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 415, in request\n    return self.rest_client.DELETE(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 270, in DELETE\n    return self.request("DELETE", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.22s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-gspo-glm47-combi-63111-v2/default/.gitattributes from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 0.63s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_63111_v2 status is now failed due to DeploymentManager action
chaiml-gspo-glm47-combi_63111_v2 status is now torndown due to DeploymentManager action