Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name zai-org-glm-5-v1-uploader
Waiting for job on zai-org-glm-5-v1-uploader to finish
zai-org-glm-5-v1-uploader: Using quantization_mode: none
zai-org-glm-5-v1-uploader: Downloading snapshot of zai-org/GLM-5...
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
zai-org-glm-5-v1-uploader: Downloaded in 549.313s
Job zai-org-glm-5-v1-uploader completed after 1224.26s with status: failed
Job failed zai-org-glm-5-v1-uploader:
Stopping job with name zai-org-glm-5-v1-uploader
%s, retrying in %s seconds...
Starting job with name zai-org-glm-5-v1-uploader
Waiting for job on zai-org-glm-5-v1-uploader to finish
Failed to get response for submission chaiml-reward-dpo-434a-_51521_v3: ('http://chaiml-reward-dpo-434a-51521-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
zai-org-glm-5-v1-uploader: Using quantization_mode: none
zai-org-glm-5-v1-uploader: Downloading snapshot of zai-org/GLM-5...
Failed to get response for submission chaiml-reward-dpo-434a-_51521_v3: HTTPConnectionPool(host='chaiml-reward-dpo-434a-51521-v3-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
zai-org-glm-5-v1-uploader: Downloaded in 553.972s
Job zai-org-glm-5-v1-uploader completed after 1205.63s with status: failed
Job failed zai-org-glm-5-v1-uploader:
Stopping job with name zai-org-glm-5-v1-uploader
%s, retrying in %s seconds...
Starting job with name zai-org-glm-5-v1-uploader
Waiting for job on zai-org-glm-5-v1-uploader to finish
zai-org-glm-5-v1-uploader: Using quantization_mode: none
zai-org-glm-5-v1-uploader: Downloading snapshot of zai-org/GLM-5...
zai-org-glm-5-v1-uploader: Downloaded in 505.713s
Job zai-org-glm-5-v1-uploader completed after 1183.69s with status: failed
Job failed zai-org-glm-5-v1-uploader:
Stopping job with name zai-org-glm-5-v1-uploader
clean up pipeline due to error=VLLMUploaderError('')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.18s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Pipeline stage VLLMModelDeleter completed in 0.21s
Shutdown handler de-registered
zai-org-glm-5_v1 status is now failed due to DeploymentManager action
zai-org-glm-5_v1 status is now torndown due to DeploymentManager action