Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
chaiml-glm-air-4-5-sft-88817-v1-uploader: Using quantization_mode: none
Starting job with name chaiml-glm-air-4-5-sft-82160-v1-uploader
chaiml-glm-air-4-5-sft-88817-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v2_e1...
Waiting for job on chaiml-glm-air-4-5-sft-82160-v1-uploader to finish
chaiml-glm-air-4-5-sft-62396-v1-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-62396-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v1_e2...
chaiml-glm-air-4-5-sft-82160-v1-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-82160-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v1_e1...
chaiml-glm-air-4-5-sft-81424-v1-uploader: Downloaded in 72.370s
chaiml-glm-air-4-5-sft-88817-v1-uploader: Downloaded in 87.055s
chaiml-glm-air-4-5-sft-62396-v1-uploader: Downloaded in 87.705s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Retrying (%r) after connection broken by '%r': %s
%s, retrying in %s seconds...
chaiml-glm-air-4-5-sft-82160-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v1_e1 in 168.013s
chaiml-glm-air-4-5-sft-82160-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-81424-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v2_e2 in 153.349s
chaiml-glm-air-4-5-sft-82160-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-81424-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-82160-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-62396-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v1_e2 in 165.419s
chaiml-glm-air-4-5-sft-81424-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-82160-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-62396-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-81424-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-82160-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-62396-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-81424-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-82160-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-81424-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-82160-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-62396-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-81424-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-82160-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-81424-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-82160-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-62396-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-81424-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-82160-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-81424-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-82160-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-88817-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v2_e1 in 166.222s
chaiml-glm-air-4-5-sft-62396-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-81424-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-82160-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-88817-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-62396-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-81424-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-82160-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-88817-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-81424-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-82160-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-88817-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-62396-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-81424-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-82160-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-88817-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-62396-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-81424-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-82160-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-88817-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-62396-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-81424-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-82160-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-88817-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-81424-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-82160-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-88817-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-62396-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-81424-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-82160-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default
chaiml-glm-air-4-5-sft-88817-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-81424-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-88817-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-62396-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-81424-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-88817-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/README.md
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/args.json
chaiml-glm-air-4-5-sft-88817-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-62396-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/config.json
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/README.md
chaiml-glm-air-4-5-sft-88817-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/README.md
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-88817-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/args.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/args.json
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/config.json
chaiml-glm-air-4-5-sft-88817-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-88817-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-88817-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/config.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-88817-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-88817-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-88817-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/args.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/config.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/README.md
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-81424-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-81424-v1/default/model-00030-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-81424-v1-uploader completed after 281.0s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-81424-v1-uploader
Pipeline stage VLLMUploader completed in 297.51s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.43s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-81424-v1
Waiting for inference service chaiml-glm-air-4-5-sft-81424-v1 to be ready
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-88817-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-88817-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00003-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-88817-v1-uploader completed after 476.52s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-88817-v1-uploader
Pipeline stage VLLMUploader completed in 478.90s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.24s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-88817-v1
Waiting for inference service chaiml-glm-air-4-5-sft-88817-v1 to be ready
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-82160-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-82160-v1/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v1/default/model-00024-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-82160-v1-uploader completed after 502.34s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-82160-v1-uploader
Pipeline stage VLLMUploader completed in 505.55s
Job chaiml-glm-air-4-5-sft-62396-v1-uploader completed after 510.28s with status: succeeded
run pipeline stage %s
Stopping job with name chaiml-glm-air-4-5-sft-62396-v1-uploader
Running pipeline stage VLLMTemplater
Pipeline stage VLLMUploader completed in 513.16s
run pipeline stage %s
Pipeline stage VLLMTemplater completed in 2.07s
Running pipeline stage VLLMTemplater
run pipeline stage %s
Running pipeline stage VLLMDeployer
Pipeline stage VLLMTemplater completed in 1.32s
Creating inference service chaiml-glm-air-4-5-sft-82160-v1
run pipeline stage %s
Running pipeline stage VLLMDeployer
Waiting for inference service chaiml-glm-air-4-5-sft-82160-v1 to be ready
Creating inference service chaiml-glm-air-4-5-sft-62396-v1
Waiting for inference service chaiml-glm-air-4-5-sft-62396-v1 to be ready
Inference service chaiml-glm-air-4-5-sft-81424-v1 ready after 721.8596274852753s
Pipeline stage VLLMDeployer completed in 725.51s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.0899243354797363s
Received healthy response to inference request in 2.270348072052002s
Received healthy response to inference request in 2.4400196075439453s
Received healthy response to inference request in 2.4915225505828857s
Received healthy response to inference request in 2.2658448219299316s
Received healthy response to inference request in 2.3208601474761963s
Received healthy response to inference request in 2.4540395736694336s
Received healthy response to inference request in 2.59826922416687s
Received healthy response to inference request in 2.3536391258239746s
Received healthy response to inference request in 3.1978797912597656s
Received healthy response to inference request in 2.425159454345703s
Received healthy response to inference request in 2.293855667114258s
Received healthy response to inference request in 2.405691623687744s
Received healthy response to inference request in 2.64884614944458s
Received healthy response to inference request in 2.1335482597351074s
Received healthy response to inference request in 2.303635597229004s
Received healthy response to inference request in 2.117995500564575s
Received healthy response to inference request in 2.3345229625701904s
Received healthy response to inference request in 2.1718997955322266s
Received healthy response to inference request in 2.1411781311035156s
Received healthy response to inference request in 2.7286136150360107s
Received healthy response to inference request in 2.395423650741577s
Received healthy response to inference request in 2.374554395675659s
Received healthy response to inference request in 2.3797173500061035s
Received healthy response to inference request in 2.302260637283325s
Received healthy response to inference request in 2.235417127609253s
Received healthy response to inference request in 2.256667137145996s
Received healthy response to inference request in 2.378697633743286s
Received healthy response to inference request in 2.2370731830596924s
Received healthy response to inference request in 2.267700672149658s
30 requests
0 failed requests
5th percentile: 2.136981701850891
10th percentile: 2.1688276290893556
20th percentile: 2.2527483463287354
30th percentile: 2.269553852081299
40th percentile: 2.3030856132507322
50th percentile: 2.3440810441970825
60th percentile: 2.3791055202484133
70th percentile: 2.4115319728851317
80th percentile: 2.461536169052124
90th percentile: 2.6568228960037232
95th percentile: 2.927334511280059
99th percentile: 3.166572709083557
mean time: 2.40049352645874
Pipeline stage StressChecker completed in 108.69s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.64s
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-_81424_v1 status is now deployed due to DeploymentManager action
Inference service chaiml-glm-air-4-5-sft-88817-v1 ready after 740.4054746627808s
Pipeline stage VLLMDeployer completed in 743.42s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.1605920791625977s
Received healthy response to inference request in 2.1072463989257812s
Inference service chaiml-glm-air-4-5-sft-82160-v1 ready after 710.5436074733734s
Received healthy response to inference request in 2.2162249088287354s
Pipeline stage VLLMDeployer completed in 715.23s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.5157859325408936s
Received healthy response to inference request in 2.1790802478790283s
Received healthy response to inference request in 2.172614812850952s
Received healthy response to inference request in 2.2543277740478516s
Received healthy response to inference request in 2.2092514038085938s
Received healthy response to inference request in 2.35961651802063s
Received healthy response to inference request in 2.315986394882202s
Received healthy response to inference request in 2.110222339630127s
Received healthy response to inference request in 3.132758378982544s
Received healthy response to inference request in 2.5911412239074707s
Received healthy response to inference request in 2.4582316875457764s
Inference service chaiml-glm-air-4-5-sft-62396-v1 ready after 729.0929281711578s
Received healthy response to inference request in 2.3368616104125977s
Pipeline stage VLLMDeployer completed in 732.23s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2385261058807373s
Received healthy response to inference request in 2.1787636280059814s
Received healthy response to inference request in 2.252835512161255s
Received healthy response to inference request in 2.2578535079956055s
Received healthy response to inference request in 2.51794695854187s
Received healthy response to inference request in 2.1755120754241943s
Received healthy response to inference request in 2.4581363201141357s
Received healthy response to inference request in 3.373558759689331s
Received healthy response to inference request in 2.384477376937866s
Received healthy response to inference request in 2.7047150135040283s
Received healthy response to inference request in 2.511580228805542s
Received healthy response to inference request in 2.2953388690948486s
Received healthy response to inference request in 2.189657688140869s
Received healthy response to inference request in 2.427865982055664s
Received healthy response to inference request in 2.218083381652832s
Received healthy response to inference request in 2.522264003753662s
Received healthy response to inference request in 2.323897361755371s
Received healthy response to inference request in 2.4363081455230713s
Received healthy response to inference request in 2.3793137073516846s
Received healthy response to inference request in 2.337404489517212s
Received healthy response to inference request in 2.221872091293335s
Received healthy response to inference request in 2.288987874984741s
Received healthy response to inference request in 2.419807195663452s
Received healthy response to inference request in 2.5317070484161377s
Received healthy response to inference request in 2.907963752746582s
Received healthy response to inference request in 2.1919291019439697s
Received healthy response to inference request in 3.3375890254974365s
Received healthy response to inference request in 2.33583402633667s
Received healthy response to inference request in 2.3622536659240723s
Received healthy response to inference request in 2.159315824508667s
Received healthy response to inference request in 2.3301939964294434s
Received healthy response to inference request in 2.3213229179382324s
Received healthy response to inference request in 2.361079692840576s
Received healthy response to inference request in 2.2999107837677s
Received healthy response to inference request in 3.2827351093292236s
Received healthy response to inference request in 2.69809627532959s
Received healthy response to inference request in 2.495739221572876s
Received healthy response to inference request in 2.948634386062622s
Received healthy response to inference request in 2.151737928390503s
Received healthy response to inference request in 2.211366891860962s
Received healthy response to inference request in 2.324903726577759s
Received healthy response to inference request in 2.152817726135254s
Received healthy response to inference request in 2.1559388637542725s
Received healthy response to inference request in 2.2838656902313232s
Received healthy response to inference request in 2.324925184249878s
Received healthy response to inference request in 2.204179525375366s
Received healthy response to inference request in 2.490138053894043s
Received healthy response to inference request in 2.9081273078918457s
Received healthy response to inference request in 2.427952766418457s
Received healthy response to inference request in 2.446955919265747s
Received healthy response to inference request in 2.3845417499542236s
Received healthy response to inference request in 2.2183456420898438s
Received healthy response to inference request in 2.5549240112304688s
Received healthy response to inference request in 2.3834614753723145s
Received healthy response to inference request in 2.3536202907562256s
Received healthy response to inference request in 2.696758985519409s
Received healthy response to inference request in 2.913245916366577s
Received healthy response to inference request in 2.5243897438049316s
Received healthy response to inference request in 2.3131706714630127s
Received healthy response to inference request in 2.2004506587982178s
Received healthy response to inference request in 2.468712329864502s
30 requests
0 failed requests
5th percentile: 2.152223837375641
10th percentile: 2.1586660146713257
Received healthy response to inference request in 2.2159996032714844s
Received healthy response to inference request in 2.180129051208496s
20th percentile: 2.202503538131714
30th percentile: 2.220735478401184
40th percentile: 2.2803447246551514
50th percentile: 2.32045578956604
60th percentile: 2.405209684371948
Received healthy response to inference request in 2.330249786376953s
Received healthy response to inference request in 2.202254056930542s
70th percentile: 2.4678035974502563
80th percentile: 2.536350440979004
90th percentile: 2.7415624856948857
95th percentile: 3.1480669140815736
Received healthy response to inference request in 2.3090391159057617s
Received healthy response to inference request in 2.1526732444763184s
99th percentile: 3.286259911060333
mean time: 2.4245293299357096
Pipeline stage StressChecker completed in 106.74s
run pipeline stage %s
Received healthy response to inference request in 2.344700813293457s
Received healthy response to inference request in 2.1544501781463623s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
30 requests
run_pipeline:run_in_cloud %s
0 failed requests
starting trigger_guanaco_pipeline args=%s
5th percentile: 2.153472864627838
10th percentile: 2.1763322830200194
Received healthy response to inference request in 2.194460391998291s
triggered trigger_guanaco_pipeline args=%s
20th percentile: 2.189569091796875
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.86s
30th percentile: 2.2750043153762816
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-_88817_v1 status is now deployed due to DeploymentManager action
50th percentile: 2.348510503768921
60th percentile: 2.383893585205078
Received healthy response to inference request in 2.5856378078460693s
70th percentile: 2.4401198863983153
80th percentile: 2.5325858116149904
90th percentile: 2.9167847633361816
95th percentile: 3.132389783859252
99th percentile: 3.3472199010849
Received healthy response to inference request in 2.2151243686676025s
mean time: 2.4461771408716837
Pipeline stage StressChecker completed in 106.40s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Received healthy response to inference request in 2.3096923828125s
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.68s
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-_82160_v1 status is now deployed due to DeploymentManager action
chaiml-glm-air-4-5-sft-_82160_v1 status is now inactive due to auto deactivation removed underperforming models