Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-0ef6-85585-v1-uploader
Waiting for job on chaiml-reward-dpo-0ef6-85585-v1-uploader to finish
chaiml-glm-air-4-5-sft-92345-v3-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-92345-v3-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_mid_lr_e2...
chaiml-reward-dpo-0ef6-85585-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-0ef6-85585-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-0ef6-chaiml-glm-air-4-5-sft-_92345_v1...
chaiml-glm-air-4-5-sft-92345-v3-uploader: Downloaded in 71.136s
chaiml-reward-dpo-0ef6-85585-v1-uploader: Downloaded in 69.856s
chaiml-glm-air-4-5-sft-92345-v3-uploader: Processed model ChaiML/glm_air_4_5_sft_mid_lr_e2 in 152.856s
chaiml-glm-air-4-5-sft-92345-v3-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-92345-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v3-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-92345-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-92345-v3-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-92345-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v3-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-92345-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v3-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-92345-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v3-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-92345-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-92345-v3-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-92345-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-92345-v3-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-92345-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-92345-v3-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-92345-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-92345-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/.gitattributes
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/tokenizer_config.json
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/config.json
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/README.md
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/args.json
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/chat_template.jinja
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/tokenizer.json
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00021-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-92345-v3-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-92345-v3/default/model-00014-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-92345-v3-uploader completed after 229.01s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-92345-v3-uploader
Pipeline stage VLLMUploader completed in 230.42s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.46s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-92345-v3
Waiting for inference service chaiml-glm-air-4-5-sft-92345-v3 to be ready
chaiml-reward-dpo-0ef6-85585-v1-uploader: Processed model ChaiML/reward-dpo-0ef6-chaiml-glm-air-4-5-sft-_92345_v1 in 225.667s
chaiml-reward-dpo-0ef6-85585-v1-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-0ef6-85585-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-0ef6-85585-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-reward-dpo-0ef6-85585-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-reward-dpo-0ef6-85585-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-reward-dpo-0ef6-85585-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-0ef6-85585-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-0ef6-85585-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-0ef6-85585-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-0ef6-85585-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-0ef6-85585-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-0ef6-85585-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-0ef6-85585-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-0ef6-85585-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-0ef6-85585-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-0ef6-85585-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-reward-dpo-0ef6-85585-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-reward-dpo-0ef6-85585-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-0ef6-85585-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/.gitattributes
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/README.md
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/chat_template.jinja
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/special_tokens_map.json
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/tokenizer_config.json
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/args.json
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/config.json
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model.safetensors.index.json
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/tokenizer.json
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00043-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00003-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00020-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00001-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00012-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00004-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00025-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00015-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00019-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00028-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00018-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00008-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00026-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00010-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00024-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00034-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00016-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00023-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00036-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00014-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00022-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00038-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00009-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00002-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00030-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00035-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00033-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00011-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00032-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00031-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00037-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00007-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00013-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00021-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00039-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-0ef6-85585-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-0ef6-85585-v1/default/model-00017-of-00043.safetensors
Job chaiml-reward-dpo-0ef6-85585-v1-uploader completed after 322.18s with status: succeeded
Stopping job with name chaiml-reward-dpo-0ef6-85585-v1-uploader
Pipeline stage VLLMUploader completed in 324.03s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.55s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-reward-dpo-0ef6-85585-v1
Waiting for inference service chaiml-reward-dpo-0ef6-85585-v1 to be ready
Inference service chaiml-glm-air-4-5-sft-92345-v3 ready after 556.5346937179565s
Pipeline stage VLLMDeployer completed in 558.19s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.9660706520080566s
Received healthy response to inference request in 2.4762842655181885s
Received healthy response to inference request in 2.4649441242218018s
Received healthy response to inference request in 2.4618139266967773s
Received healthy response to inference request in 2.2812552452087402s
Received healthy response to inference request in 2.2987217903137207s
Received healthy response to inference request in 2.222438097000122s
Received healthy response to inference request in 2.336191177368164s
Received healthy response to inference request in 2.0653042793273926s
Received healthy response to inference request in 3.1847305297851562s
Received healthy response to inference request in 3.323333501815796s
Received healthy response to inference request in 2.1129701137542725s
Received healthy response to inference request in 2.253965377807617s
Received healthy response to inference request in 2.346097946166992s
Received healthy response to inference request in 2.2611260414123535s
Received healthy response to inference request in 2.5303688049316406s
Received healthy response to inference request in 2.46460223197937s
Received healthy response to inference request in 2.1606521606445312s
Received healthy response to inference request in 2.153907060623169s
Received healthy response to inference request in 2.08660888671875s
Received healthy response to inference request in 2.5124447345733643s
Received healthy response to inference request in 2.2774455547332764s
Received healthy response to inference request in 2.1916329860687256s
Received healthy response to inference request in 2.1116998195648193s
Received healthy response to inference request in 2.190089702606201s
Received healthy response to inference request in 2.291569471359253s
Received healthy response to inference request in 2.1719319820404053s
Received healthy response to inference request in 2.097409248352051s
Received healthy response to inference request in 2.085055351257324s
Received healthy response to inference request in 2.0665042400360107s
30 requests
0 failed requests
5th percentile: 2.074852240085602
10th percentile: 2.0864535331726075
20th percentile: 2.112716054916382
30th percentile: 2.168548035621643
40th percentile: 2.2101160526275634
50th percentile: 2.269285798072815
60th percentile: 2.29443039894104
70th percentile: 2.3808127403259274
80th percentile: 2.4672121524810793
90th percentile: 2.573938989639283
95th percentile: 3.0863335847854607
99th percentile: 3.2831386399269107
mean time: 2.348238976796468
Pipeline stage StressChecker completed in 86.48s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.89s
Shutdown handler de-registered
chaiml-glm-air-4-5-sft-_92345_v3 status is now deployed due to DeploymentManager action
Inference service chaiml-reward-dpo-0ef6-85585-v1 ready after 565.7455613613129s
Pipeline stage VLLMDeployer completed in 567.43s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.3287813663482666s
Received healthy response to inference request in 2.3009519577026367s
Received healthy response to inference request in 2.121821165084839s
Received healthy response to inference request in 2.176042079925537s
Received healthy response to inference request in 2.2289702892303467s
Received healthy response to inference request in 2.794609308242798s
Received healthy response to inference request in 2.242356061935425s
Received healthy response to inference request in 2.205845594406128s
Received healthy response to inference request in 2.2668983936309814s
Received healthy response to inference request in 2.1980669498443604s
Received healthy response to inference request in 2.0816187858581543s
Received healthy response to inference request in 2.190028190612793s
Received healthy response to inference request in 2.366330862045288s
Received healthy response to inference request in 2.675441026687622s
Received healthy response to inference request in 2.065809488296509s
Received healthy response to inference request in 2.2241315841674805s
Received healthy response to inference request in 3.1758768558502197s
Received healthy response to inference request in 2.2429797649383545s
Received healthy response to inference request in 2.916245698928833s
Received healthy response to inference request in 2.1539857387542725s
Received healthy response to inference request in 2.090789556503296s
Received healthy response to inference request in 2.0513365268707275s
Received healthy response to inference request in 3.0872092247009277s
Received healthy response to inference request in 2.0899319648742676s
Received healthy response to inference request in 2.20695161819458s
Received healthy response to inference request in 2.0999484062194824s
Received healthy response to inference request in 2.5776491165161133s
Received healthy response to inference request in 2.316824197769165s
Received healthy response to inference request in 2.1090822219848633s
Received healthy response to inference request in 2.0839972496032715s
30 requests
0 failed requests
5th percentile: 2.0729236721992494
10th percentile: 2.08375940322876
20th percentile: 2.098116636276245
30th percentile: 2.1443363666534423
40th percentile: 2.194851446151733
50th percentile: 2.2155416011810303
60th percentile: 2.2426055431365968
70th percentile: 2.3057136297225953
80th percentile: 2.408594512939454
90th percentile: 2.8067729473114014
95th percentile: 3.0102756381034848
99th percentile: 3.1501632428169253
mean time: 2.3223503748575847
Pipeline stage StressChecker completed in 79.14s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.53s
Shutdown handler de-registered
chaiml-reward-dpo-0ef6-_85585_v1 status is now deployed due to DeploymentManager action
chaiml-reward-dpo-0ef6-_85585_v1 status is now inactive due to auto deactivation removed underperforming models