Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-4-5-sft-51822-v4-uploader
Waiting for job on chaiml-glm-air-4-5-sft-51822-v4-uploader to finish
chaiml-glm-air-4-5-sft-51822-v4-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-51822-v4-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_lower_lr_e2...
chaiml-glm-air-4-5-sft-51822-v4-uploader: Downloaded in 71.519s
chaiml-glm-air-4-5-sft-51822-v4-uploader: Processed model ChaiML/glm_air_4_5_sft_lower_lr_e2 in 149.332s
chaiml-glm-air-4-5-sft-51822-v4-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-51822-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v4-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-51822-v4-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-51822-v4-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-51822-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v4-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-51822-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v4-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-51822-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v4-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-51822-v4-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-51822-v4-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-51822-v4-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-51822-v4-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-51822-v4-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-51822-v4-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-51822-v4-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-51822-v4-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/chat_template.jinja
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/config.json
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/args.json
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/README.md
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/.gitattributes
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-51822-v4-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-51822-v4/default/model-00037-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-51822-v4-uploader completed after 218.0s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-51822-v4-uploader
Pipeline stage VLLMUploader completed in 219.21s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.29s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-51822-v4
Waiting for inference service chaiml-glm-air-4-5-sft-51822-v4 to be ready
Inference service chaiml-glm-air-4-5-sft-51822-v4 ready after 486.41857981681824s
Pipeline stage VLLMDeployer completed in 489.13s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.922823190689087s
Received healthy response to inference request in 2.1617801189422607s
Received healthy response to inference request in 2.1283040046691895s
Received healthy response to inference request in 2.2244033813476562s
Received healthy response to inference request in 2.2321090698242188s
Received healthy response to inference request in 2.192814350128174s
Received healthy response to inference request in 2.2619028091430664s
Received healthy response to inference request in 2.4709479808807373s
Received healthy response to inference request in 2.201702117919922s
Received healthy response to inference request in 2.210392475128174s
Received healthy response to inference request in 2.1951425075531006s
Received healthy response to inference request in 2.223024606704712s
Received healthy response to inference request in 2.2665340900421143s
Received healthy response to inference request in 2.2566871643066406s
Received healthy response to inference request in 2.3693015575408936s
Received healthy response to inference request in 2.121685743331909s
Received healthy response to inference request in 2.2153005599975586s
Received healthy response to inference request in 2.142703056335449s
Received healthy response to inference request in 2.209739923477173s
Received healthy response to inference request in 2.0207676887512207s
Received healthy response to inference request in 2.1005606651306152s
Received healthy response to inference request in 2.2273476123809814s
Received healthy response to inference request in 2.1686503887176514s
Received healthy response to inference request in 2.095198392868042s
Received healthy response to inference request in 2.131951332092285s
Received healthy response to inference request in 2.333880662918091s
Received healthy response to inference request in 2.3849921226501465s
Received healthy response to inference request in 2.144540786743164s
Received healthy response to inference request in 2.150320291519165s
Received healthy response to inference request in 2.383113384246826s
30 requests
0 failed requests
5th percentile: 2.0976114153861998
10th percentile: 2.11957323551178
20th percentile: 2.1405527114868166
30th percentile: 2.158342170715332
40th percentile: 2.19421124458313
50th percentile: 2.2100661993026733
60th percentile: 2.2235761165618895
70th percentile: 2.2394824981689454
80th percentile: 2.2800034046173097
90th percentile: 2.383301258087158
95th percentile: 2.432267844676971
99th percentile: 2.791779379844666
mean time: 2.238287401199341
Pipeline stage StressChecker completed in 73.19s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.18s
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-_51822_v4 status is now deployed due to DeploymentManager action
chaiml-glm-air-4-5-sft-_51822_v4 status is now inactive due to auto deactivation removed underperforming models
chaiml-glm-air-4-5-sft-_51822_v4 status is now torndown due to DeploymentManager action