Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-bobo-swi-99451-v1-uploader
Waiting for job on chaiml-glm-air-bobo-swi-99451-v1-uploader to finish
chaiml-glm-air-bobo-swi-99451-v1-uploader: Using quantization_mode: none
chaiml-glm-air-bobo-swi-99451-v1-uploader: Downloading snapshot of ChaiML/glm_air_bobo_swim_v1-step687-merged...
chaiml-glm-air-bobo-swi-99451-v1-uploader: Downloaded in 73.491s
chaiml-glm-air-bobo-swi-99451-v1-uploader: Processed model ChaiML/glm_air_bobo_swim_v1-step687-merged in 151.162s
chaiml-glm-air-bobo-swi-99451-v1-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-bobo-swi-99451-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-bobo-swi-99451-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-bobo-swi-99451-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-bobo-swi-99451-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-bobo-swi-99451-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-bobo-swi-99451-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-bobo-swi-99451-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-bobo-swi-99451-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-bobo-swi-99451-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-bobo-swi-99451-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-bobo-swi-99451-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-bobo-swi-99451-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-bobo-swi-99451-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-bobo-swi-99451-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-bobo-swi-99451-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-bobo-swi-99451-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-bobo-swi-99451-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-bobo-swi-99451-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/.gitattributes
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/chat_template.jinja
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/config.json
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/special_tokens_map.json
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/README.md
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/generation_config.json
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/tokenizer_config.json
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model.safetensors.index.json
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/tokenizer.json
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00020-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00013-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00039-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00031-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00012-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00010-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00004-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00021-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00019-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00009-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00032-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00025-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00042-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00023-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00029-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00006-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00001-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00038-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00024-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00016-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00026-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00037-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00018-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00002-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00028-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00033-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00003-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00027-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00005-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00030-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00008-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00034-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00035-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00007-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00022-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00017-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00011-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00041-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00040-of-00043.safetensors
chaiml-glm-air-bobo-swi-99451-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-bobo-swi-99451-v1/default/model-00015-of-00043.safetensors
Job chaiml-glm-air-bobo-swi-99451-v1-uploader completed after 227.05s with status: succeeded
Stopping job with name chaiml-glm-air-bobo-swi-99451-v1-uploader
Pipeline stage VLLMUploader completed in 228.26s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.56s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-bobo-swi-99451-v1
Waiting for inference service chaiml-glm-air-bobo-swi-99451-v1 to be ready
Inference service chaiml-glm-air-bobo-swi-99451-v1 ready after 304.2834258079529s
Pipeline stage VLLMDeployer completed in 305.74s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.398667097091675s
Received healthy response to inference request in 2.016608238220215s
Received healthy response to inference request in 1.8576931953430176s
Received healthy response to inference request in 1.8052582740783691s
Received healthy response to inference request in 2.116133213043213s
Received healthy response to inference request in 3.3515090942382812s
Received healthy response to inference request in 1.9411256313323975s
Received healthy response to inference request in 2.003840684890747s
Received healthy response to inference request in 2.060194969177246s
Received healthy response to inference request in 1.8524794578552246s
Received healthy response to inference request in 1.9334022998809814s
Received healthy response to inference request in 1.8274192810058594s
Received healthy response to inference request in 1.8991761207580566s
Received healthy response to inference request in 1.8953728675842285s
Received healthy response to inference request in 1.8762578964233398s
Received healthy response to inference request in 2.327029228210449s
Received healthy response to inference request in 1.8920857906341553s
Received healthy response to inference request in 2.0633866786956787s
Received healthy response to inference request in 2.020237684249878s
Received healthy response to inference request in 1.7574551105499268s
Received healthy response to inference request in 2.116152286529541s
Received healthy response to inference request in 2.190422296524048s
Received healthy response to inference request in 2.022874355316162s
Received healthy response to inference request in 1.9103870391845703s
Received healthy response to inference request in 1.7657151222229004s
Received healthy response to inference request in 1.7877013683319092s
Received healthy response to inference request in 1.8842172622680664s
Received healthy response to inference request in 1.890859603881836s
Received healthy response to inference request in 2.0263819694519043s
Received healthy response to inference request in 1.8709328174591064s
30 requests
0 failed requests
5th percentile: 1.7756089329719544
10th percentile: 1.803502583503723
20th percentile: 1.856650447845459
30th percentile: 1.8818294525146484
40th percentile: 1.8940580368041993
50th percentile: 1.9218946695327759
60th percentile: 2.0089477062225343
70th percentile: 2.0239266395568847
80th percentile: 2.0739359855651855
90th percentile: 2.2040829896926883
95th percentile: 2.366430056095123
99th percentile: 3.0751849150657664
mean time: 2.0120325644810992
Pipeline stage StressChecker completed in 66.50s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.14s
Shutdown handler de-registered
chaiml-glm-air-bobo-swi_99451_v1 status is now deployed due to DeploymentManager action
chaiml-glm-air-bobo-swi_99451_v1 status is now inactive due to auto deactivation removed underperforming models