Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-4-5-sft-27302-v1-uploader
Waiting for job on chaiml-glm-air-4-5-sft-27302-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-4-5-sft-51822-v1-uploader
Waiting for job on chaiml-glm-air-4-5-sft-51822-v1-uploader to finish
chaiml-glm-air-4-5-sft-27302-v1-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-27302-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_lower_lr_e3...
chaiml-glm-air-4-5-sft-51822-v1-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-51822-v1-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_lower_lr_e2...
chaiml-glm-air-4-5-sft-27302-v1-uploader: Downloaded in 75.942s
chaiml-glm-air-4-5-sft-51822-v1-uploader: Downloaded in 73.034s
chaiml-glm-air-4-5-sft-51822-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_lower_lr_e2 in 149.504s
chaiml-glm-air-4-5-sft-51822-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-51822-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-51822-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-51822-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-51822-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-51822-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-51822-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-51822-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-51822-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-51822-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-51822-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-51822-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-51822-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-51822-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/args.json
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/config.json
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/README.md
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-27302-v1-uploader: Processed model ChaiML/glm_air_4_5_sft_lower_lr_e3 in 154.490s
chaiml-glm-air-4-5-sft-27302-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-27302-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-27302-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-27302-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-27302-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-27302-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-27302-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-27302-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-27302-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-27302-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-27302-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-27302-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-27302-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-27302-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-27302-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-27302-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-27302-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-27302-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-27302-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/args.json
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/.gitattributes
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/config.json
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/chat_template.jinja
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/README.md
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/tokenizer.json
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-27302-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-27302-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v1/default/model-00007-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-27302-v1-uploader completed after 225.49s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-27302-v1-uploader
Pipeline stage VLLMUploader completed in 226.83s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.43s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-27302-v1
Waiting for inference service chaiml-glm-air-4-5-sft-27302-v1 to be ready
Job chaiml-glm-air-4-5-sft-51822-v1-uploader completed after 227.66s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-51822-v1-uploader
Pipeline stage VLLMUploader completed in 229.16s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.41s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-51822-v1
Waiting for inference service chaiml-glm-air-4-5-sft-51822-v1 to be ready
Inference service chaiml-glm-air-4-5-sft-27302-v1 ready after 546.5355668067932s
Pipeline stage VLLMDeployer completed in 547.90s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.055669069290161s
Received healthy response to inference request in 2.178123712539673s
Received healthy response to inference request in 2.198747158050537s
Received healthy response to inference request in 2.295241117477417s
Received healthy response to inference request in 2.106165885925293s
Received healthy response to inference request in 2.4550423622131348s
Received healthy response to inference request in 2.225661277770996s
Received healthy response to inference request in 2.173138380050659s
Received healthy response to inference request in 2.2117512226104736s
Received healthy response to inference request in 2.5066962242126465s
Received healthy response to inference request in 2.073866844177246s
Received healthy response to inference request in 2.175349235534668s
Received healthy response to inference request in 2.1177706718444824s
Received healthy response to inference request in 2.237788677215576s
Received healthy response to inference request in 2.2272145748138428s
Received healthy response to inference request in 2.520368814468384s
Received healthy response to inference request in 2.197733163833618s
Received healthy response to inference request in 2.3395259380340576s
Received healthy response to inference request in 2.166050672531128s
Received healthy response to inference request in 2.009427547454834s
Received healthy response to inference request in 2.150223731994629s
Received healthy response to inference request in 2.1230013370513916s
Received healthy response to inference request in 2.0330684185028076s
Received healthy response to inference request in 2.2682015895843506s
Received healthy response to inference request in 2.0909388065338135s
Received healthy response to inference request in 2.134093999862671s
Received healthy response to inference request in 2.133218765258789s
Received healthy response to inference request in 2.688910961151123s
Received healthy response to inference request in 2.2522289752960205s
Received healthy response to inference request in 2.234869956970215s
30 requests
0 failed requests
5th percentile: 2.0514277100563048
10th percentile: 2.0892316102981567
20th percentile: 2.1219552040100096
30th percentile: 2.1453848123550414
40th percentile: 2.1744648933410646
50th percentile: 2.1982401609420776
60th percentile: 2.2262825965881348
70th percentile: 2.2421207666397094
80th percentile: 2.3040980815887453
90th percentile: 2.50806348323822
95th percentile: 2.61306699514389
99th percentile: 2.9493092179298404
mean time: 2.252669636408488
Pipeline stage StressChecker completed in 81.01s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.61s
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-_27302_v1 status is now deployed due to DeploymentManager action
chaiml-glm-air-4-5-sft-_27302_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-glm-air-4-5-sft-_27302_v1 status is now torndown due to DeploymentManager action