Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader
Waiting for job on chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader
Waiting for job on chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader to finish
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_lr1_e4...
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_lr1_e1...
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: Downloaded in 68.401s
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: Downloaded in 71.114s
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_lr1_e4 in 147.818s
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_lr1_e1 in 149.531s
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/README.md
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/args.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/config.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/README.md
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/args.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/config.json
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e4-v1/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-lr1-e1-v1/default/model-00021-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader completed after 249.76s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-lr1-e4-v1-uploader
Pipeline stage VLLMUploader completed in 253.51s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Job chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader completed after 245.94s with status: succeeded
Pipeline stage VLLMTemplater completed in 1.66s
Stopping job with name chaiml-glm-air-4-5-sft-lr1-e1-v1-uploader
run pipeline stage %s
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploader completed in 251.31s
Creating inference service chaiml-glm-air-4-5-sft-lr1-e4-v1
run pipeline stage %s
Running pipeline stage VLLMTemplater
Waiting for inference service chaiml-glm-air-4-5-sft-lr1-e4-v1 to be ready
Pipeline stage VLLMTemplater completed in 14.75s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-lr1-e1-v1
Waiting for inference service chaiml-glm-air-4-5-sft-lr1-e1-v1 to be ready
Inference service chaiml-glm-air-4-5-sft-lr1-e4-v1 ready after 448.5564749240875s
Pipeline stage VLLMDeployer completed in 465.54s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.8254945278167725s
Received healthy response to inference request in 2.38220477104187s
Received healthy response to inference request in 2.3151004314422607s
Received healthy response to inference request in 2.3033080101013184s
Received healthy response to inference request in 2.1578524112701416s
Received healthy response to inference request in 2.4572439193725586s
Received healthy response to inference request in 2.447165012359619s
Received healthy response to inference request in 2.224233865737915s
Received healthy response to inference request in 2.165895938873291s
Received healthy response to inference request in 2.5468316078186035s
Received healthy response to inference request in 2.2796812057495117s
Received healthy response to inference request in 2.2249879837036133s
Received healthy response to inference request in 2.244584560394287s
Received healthy response to inference request in 2.2943832874298096s
Received healthy response to inference request in 2.2170321941375732s
Received healthy response to inference request in 2.383129835128784s
Received healthy response to inference request in 2.258295774459839s
Received healthy response to inference request in 2.1851909160614014s
Received healthy response to inference request in 2.047849178314209s
Received healthy response to inference request in 2.2471890449523926s
Received healthy response to inference request in 3.128242254257202s
Received healthy response to inference request in 2.0825893878936768s
Received healthy response to inference request in 2.0185229778289795s
Received healthy response to inference request in 2.3562259674072266s
Received healthy response to inference request in 2.053163528442383s
Received healthy response to inference request in 2.2374401092529297s
Received healthy response to inference request in 2.4301602840423584s
Received healthy response to inference request in 2.2280991077423096s
Received healthy response to inference request in 2.8198485374450684s
Received healthy response to inference request in 2.2238123416900635s
30 requests
0 failed requests
5th percentile: 2.050240635871887
10th percentile: 2.0796468019485475
20th percentile: 2.181331920623779
30th percentile: 2.2241074085235595
40th percentile: 2.2337037086486817
50th percentile: 2.2527424097061157
60th percentile: 2.297953176498413
70th percentile: 2.3640196084976197
80th percentile: 2.4335612297058105
90th percentile: 2.5741333007812504
95th percentile: 2.989465081691741
99th percentile: 3.6232913684844976
mean time: 2.3595252990722657
Pipeline stage StressChecker completed in 92.61s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 6.31s
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-lr1-e4_v1 status is now deployed due to DeploymentManager action
chaiml-glm-air-4-5-sft-lr1-e4_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-glm-air-4-5-sft-lr1-e4_v1 status is now torndown due to DeploymentManager action