Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-4-5-sft-24242-v4-uploader
Waiting for job on chaiml-glm-air-4-5-sft-24242-v4-uploader to finish
chaiml-glm-air-4-5-sft-24242-v4-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-24242-v4-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_lower_lr_e4...
chaiml-glm-air-4-5-sft-24242-v4-uploader: Downloaded in 72.243s
chaiml-glm-air-4-5-sft-24242-v4-uploader: Processed model ChaiML/glm_air_4_5_sft_lower_lr_e4 in 148.317s
chaiml-glm-air-4-5-sft-24242-v4-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-24242-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-24242-v4-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-24242-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-24242-v4-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-24242-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-24242-v4-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-24242-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-24242-v4-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-24242-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-24242-v4-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-24242-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-24242-v4-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-24242-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-24242-v4-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-24242-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-24242-v4-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-24242-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-24242-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/args.json
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/config.json
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/README.md
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/.gitattributes
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/chat_template.jinja
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/tokenizer.json
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-24242-v4-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-24242-v4/default/model-00016-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-24242-v4-uploader completed after 218.81s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-24242-v4-uploader
Pipeline stage VLLMUploader completed in 219.68s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.27s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-24242-v4
Waiting for inference service chaiml-glm-air-4-5-sft-24242-v4 to be ready
Inference service chaiml-glm-air-4-5-sft-24242-v4 ready after 465.6675446033478s
Pipeline stage VLLMDeployer completed in 466.64s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.1104273796081543s
Received healthy response to inference request in 2.18947434425354s
Received healthy response to inference request in 2.2280337810516357s
Received healthy response to inference request in 2.150495767593384s
Received healthy response to inference request in 2.289888381958008s
Received healthy response to inference request in 2.2386796474456787s
Received healthy response to inference request in 2.309918165206909s
Received healthy response to inference request in 2.194573402404785s
Received healthy response to inference request in 2.1366868019104004s
Received healthy response to inference request in 2.011993408203125s
Received healthy response to inference request in 2.800034284591675s
Received healthy response to inference request in 2.2148306369781494s
Received healthy response to inference request in 2.198730230331421s
Received healthy response to inference request in 2.0974843502044678s
Received healthy response to inference request in 2.0056779384613037s
Received healthy response to inference request in 2.1752312183380127s
Received healthy response to inference request in 2.0549771785736084s
Received healthy response to inference request in 2.3221566677093506s
Received healthy response to inference request in 2.0732474327087402s
Received healthy response to inference request in 2.2032506465911865s
Received healthy response to inference request in 2.2342236042022705s
Received healthy response to inference request in 2.2027509212493896s
Received healthy response to inference request in 2.1459298133850098s
Received healthy response to inference request in 2.333164691925049s
Received healthy response to inference request in 2.2675769329071045s
Received healthy response to inference request in 2.0684192180633545s
Received healthy response to inference request in 2.169692277908325s
Received healthy response to inference request in 2.190488815307617s
Received healthy response to inference request in 2.1022050380706787s
Received healthy response to inference request in 2.2001590728759766s
30 requests
0 failed requests
5th percentile: 2.0313361048698426
10th percentile: 2.06707501411438
20th percentile: 2.1012609004974365
30th percentile: 2.1491259813308714
40th percentile: 2.183777093887329
50th percentile: 2.196651816368103
60th percentile: 2.2029508113861085
70th percentile: 2.229890727996826
80th percentile: 2.2720392227172854
90th percentile: 2.3232574701309203
95th percentile: 2.589942967891692
99th percentile: 3.0204133820533756
mean time: 2.230680068333944
Pipeline stage StressChecker completed in 73.18s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.14s
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-_24242_v4 status is now deployed due to DeploymentManager action
chaiml-glm-air-4-5-sft-_24242_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-glm-air-4-5-sft-_24242_v4 status is now torndown due to DeploymentManager action