Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-4-5-sft-92345-v1-uploader
Waiting for job on chaiml-glm-air-4-5-sft-92345-v1-uploader to finish
chaiml-glm-air-4-5-sft-73471-v1-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-73471-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_mid_lr_e3...
chaiml-glm-air-4-5-sft-92345-v1-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-92345-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_mid_lr_e2...
chaiml-glm-air-4-5-sft-92345-v1-uploader: Downloaded in 77.209s
chaiml-glm-air-4-5-sft-73471-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_mid_lr_e3 in 154.407s
chaiml-glm-air-4-5-sft-73471-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-73471-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-73471-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-73471-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-73471-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-73471-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-73471-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-73471-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-73471-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-73471-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-73471-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-73471-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-73471-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-73471-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-73471-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-73471-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-73471-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-73471-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-73471-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/args.json
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/config.json
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/README.md
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-92345-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_mid_lr_e2 in 157.311s
chaiml-glm-air-4-5-sft-92345-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-92345-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-92345-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-92345-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-92345-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-92345-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-92345-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-92345-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-92345-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-92345-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-92345-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-92345-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-92345-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-92345-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/config.json
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/README.md
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/args.json
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v1/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00021-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-92345-v1-uploader completed after 241.54s with status: succeeded
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00035-of-00043.safetensors
Stopping job with name chaiml-glm-air-4-5-sft-92345-v1-uploader
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00020-of-00043.safetensors
Pipeline stage VLLMUploader completed in 243.13s
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00034-of-00043.safetensors
run pipeline stage %s
chaiml-glm-air-4-5-sft-73471-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-73471-v1/default/model-00038-of-00043.safetensors
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.39s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-92345-v1
Waiting for inference service chaiml-glm-air-4-5-sft-92345-v1 to be ready
Job chaiml-glm-air-4-5-sft-73471-v1-uploader completed after 258.36s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-73471-v1-uploader
Pipeline stage VLLMUploader completed in 259.69s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.38s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-73471-v1
Waiting for inference service chaiml-glm-air-4-5-sft-73471-v1 to be ready
Inference service chaiml-glm-air-4-5-sft-92345-v1 ready after 607.3457915782928s
Pipeline stage VLLMDeployer completed in 608.66s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.9064083099365234s
Received healthy response to inference request in 2.4933321475982666s
Received healthy response to inference request in 2.40625262260437s
Received healthy response to inference request in 2.1518373489379883s
Inference service chaiml-glm-air-4-5-sft-73471-v1 ready after 607.5583808422089s
Pipeline stage VLLMDeployer completed in 609.03s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3978981971740723s
Received healthy response to inference request in 2.3789682388305664s
Received healthy response to inference request in 2.364224433898926s
Received healthy response to inference request in 2.1756820678710938s
Received healthy response to inference request in 2.157297134399414s
Received healthy response to inference request in 2.2502493858337402s
Received healthy response to inference request in 2.2701146602630615s
Received healthy response to inference request in 2.175361394882202s
Received healthy response to inference request in 2.103914260864258s
Received healthy response to inference request in 2.2696051597595215s
Received healthy response to inference request in 2.2661428451538086s
Received healthy response to inference request in 2.5108296871185303s
Received healthy response to inference request in 2.248603105545044s
Received healthy response to inference request in 2.2474117279052734s
Received healthy response to inference request in 2.1871047019958496s
Received healthy response to inference request in 2.2688663005828857s
Received healthy response to inference request in 2.166888475418091s
Received healthy response to inference request in 2.125087261199951s
Received healthy response to inference request in 2.2255587577819824s
Received healthy response to inference request in 2.201908588409424s
Received healthy response to inference request in 2.444356918334961s
Received healthy response to inference request in 2.185680389404297s
Received healthy response to inference request in 2.1942405700683594s
Received healthy response to inference request in 2.4435136318206787s
Received healthy response to inference request in 2.2071025371551514s
Received healthy response to inference request in 2.325387477874756s
Received healthy response to inference request in 2.093308448791504s
Received healthy response to inference request in 2.146892786026001s
Received healthy response to inference request in 2.0775463581085205s
Received healthy response to inference request in 2.2766096591949463s
Received healthy response to inference request in 2.3509323596954346s
Received healthy response to inference request in 2.134009599685669s
Received healthy response to inference request in 2.1109461784362793s
Received healthy response to inference request in 2.2881462574005127s
Received healthy response to inference request in 3.1167502403259277s
Received healthy response to inference request in 2.1291921138763428s
Received healthy response to inference request in 2.187032461166382s
Received healthy response to inference request in 2.434760570526123s
Received healthy response to inference request in 2.109903573989868s
Received healthy response to inference request in 2.0879428386688232s
Received healthy response to inference request in 2.184422016143799s
Received healthy response to inference request in 2.244924783706665s
Received healthy response to inference request in 2.3699676990509033s
Received healthy response to inference request in 2.370659351348877s
Received healthy response to inference request in 2.1807429790496826s
Received healthy response to inference request in 2.2801060676574707s
Received healthy response to inference request in 2.3736748695373535s
Received healthy response to inference request in 2.113799810409546s
Received healthy response to inference request in 2.151092290878296s
Received healthy response to inference request in 2.123612880706787s
30 requests
Received healthy response to inference request in 2.1420040130615234s
0 failed requests
5th percentile: 2.0903573632240295
10th percentile: 2.1028536796569823
20th percentile: 2.1216502666473387
30th percentile: 2.1556591987609863
40th percentile: 2.1913862228393555
50th percentile: 2.2352417707443237
60th percentile: 2.2677315711975097
70th percentile: 2.306982088088989
Received healthy response to inference request in 2.3325212001800537s
80th percentile: 2.376107120513916
90th percentile: 2.435720205307007
95th percentile: 2.471293294429779
99th percentile: 2.7866162228584295
mean time: 2.2632385015487673
Pipeline stage StressChecker completed in 81.84s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
Received healthy response to inference request in 2.8740694522857666s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.88s
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-_92345_v1 status is now deployed due to DeploymentManager action
chaiml-glm-air-4-5-sft-_92345_v1 status is now inactive due to auto deactivation removed underperforming models