chaiml-merged-qwen-35-d_39140

submission_id: chaiml-merged-qwen-35-d_39140_v8
developer_uid: chai_backend_admin
status: torndown
model_repo: ChaiML/merged_qwen_35_dpo_lower_lr_v
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '</s>', '<|im_end|>', '###', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
timestamp: 2026-03-27T17:35:51+00:00
model_name: chaiml-merged-qwen-35-d_39140_v8
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-merged-qwen-35-d-39140-v8-uploader
Waiting for job on chaiml-merged-qwen-35-d-39140-v8-uploader to finish
chaiml-merged-qwen-35-d-39140-v8-uploader: Using quantization_mode: none
chaiml-merged-qwen-35-d-39140-v8-uploader: Downloading snapshot of ChaiML/merged_qwen_35_dpo_lower_lr_v...
chaiml-merged-qwen-35-d-39140-v8-uploader: Downloaded in 31.740s
2026-03-27T17:16:28.814889+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
chaiml-merged-qwen-35-d-39140-v8-uploader: Processed model ChaiML/merged_qwen_35_dpo_lower_lr_v in 51.452s
chaiml-merged-qwen-35-d-39140-v8-uploader: creating bucket guanaco-vllm-models
chaiml-merged-qwen-35-d-39140-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v8-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-merged-qwen-35-d-39140-v8-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-merged-qwen-35-d-39140-v8-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-merged-qwen-35-d-39140-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v8-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-d-39140-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v8-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-merged-qwen-35-d-39140-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v8-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-d-39140-v8-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-merged-qwen-35-d-39140-v8-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-merged-qwen-35-d-39140-v8-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-merged-qwen-35-d-39140-v8-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-merged-qwen-35-d-39140-v8-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-merged-qwen-35-d-39140-v8-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-merged-qwen-35-d-39140-v8-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-merged-qwen-35-d-39140-v8-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/.gitattributes
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/README.md
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/config.json
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/tokenizer_config.json
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/generation_config.json
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/model.safetensors.index.json
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/tokenizer.json
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/model-00002-of-00002.safetensors
2026-03-27T17:17:28.917927+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
chaiml-merged-qwen-35-d-39140-v8-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-merged-qwen-35-d-39140-v8/default/model-00001-of-00002.safetensors
Job chaiml-merged-qwen-35-d-39140-v8-uploader completed after 143.15s with status: succeeded
Stopping job with name chaiml-merged-qwen-35-d-39140-v8-uploader
Pipeline stage VLLMUploader completed in 143.70s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.23s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-merged-qwen-35-d-39140-v8
Waiting for inference service chaiml-merged-qwen-35-d-39140-v8 to be ready
2026-03-27T17:18:29.018556+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:19:29.120780+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:20:29.227432+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:21:29.325586+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:22:29.426638+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:23:29.527173+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:24:29.624618+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:25:29.723473+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:26:29.826850+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:27:29.927057+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:28:30.026494+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:29:30.130242+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:30:30.245469+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:31:30.347931+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:32:30.453418+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
2026-03-27T17:33:30.557822+00:00 monitor updated for chaiml-merged-qwen-35-d_39140_v8
Tearing down inference service chaiml-merged-qwen-35-d-39140-v8
clean up pipeline due to error=DeploymentError('404\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({\'Date\': \'Fri, 27 Mar 2026 17:33:39 GMT\', \'Content-Type\': \'application/json\', \'Content-Length\': \'304\', \'Connection\': \'keep-alive\', \'audit-id\': \'fdfb10cf-6457-4237-8b51-0c37d88b116e\', \'Cache-Control\': \'no-cache, private\', \'x-kubernetes-pf-flowschema-uid\': \'757159af-7462-4610-a5a7-751d984daa90\', \'x-kubernetes-pf-prioritylevel-uid\': \'5f6bf145-f133-455b-a330-c9bf3e5ea771\', \'cf-cache-status\': \'DYNAMIC\', \'set-cookie\': \'__cf_bm=gUvA7UXs_bnIdmIbwSfTk9OhRmFIutLDe8o1Arg0w3o-1774632819.4733377-1.0.1.1-JJZLELJvU3Oc8gPBsugJx2LWHBCqgDC6uCc_XH4w5isK9MYLt5hGE88K64FsOc85RWb8wHTPHT.86hMTmfG6l; HttpOnly; Secure; Path=/; Domain=coreweave.com; Expires=Fri, 27 Mar 2026 18:03:39 GMT\', \'Server\': \'cloudflare\', \'CF-RAY\': \'9e302431b898080c-IAD\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kserve.io \\\\"chaiml-merged-qwen-35-d-39140-v8\\\\" not found","reason":"NotFound","details":{"name":"chaiml-merged-qwen-35-d-39140-v8","group":"serving.kserve.io","kind":"inferenceservices"},"code":404}\\n\'\nOriginal traceback: \n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/dynamic/client.py", line 273, in request\n    api_response = self.client.call_api(\n                   ^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n                    ^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 415, in request\n    return self.rest_client.DELETE(url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 270, in DELETE\n    return self.request("DELETE", url,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request\n    raise ApiException(http_resp=r)\n')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.21s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/.gitattributes from bucket guanaco-vllm-models
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/README.md from bucket guanaco-vllm-models
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/model-00001-of-00002.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/model-00002-of-00002.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-merged-qwen-35-d-39140-v8/default/tokenizer_config.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 4.71s
Shutdown handler de-registered
chaiml-merged-qwen-35-d_39140_v8 status is now failed due to DeploymentManager action
chaiml-merged-qwen-35-d_39140_v8 status is now torndown due to DeploymentManager action