developer_uid: rirv938
submission_id: chaiml-reward-dpo-434a-_51521_v1
model_name: chaiml-reward-dpo-434a-_51521_v1
model_group: ChaiML/reward-dpo-434a-c
status: torndown
timestamp: 2026-03-08T01:22:26+00:00
num_battles: 13064
num_wins: 6941
celo_rating: 1347.86
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/reward-dpo-434a-chaiml-glm-air-4-5-sft-_62396_v1
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 9073971200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-reward-dpo-434a-_51521_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/reward-dpo-434a-chaiml-glm-air-4-5-sft-_62396_v1
model_size: 9B
ranking_group: single
us_pacific_date: 2026-03-04
win_ratio: 0.531307409675444
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', 'You:', '</s>', '###', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '[gMASK]<sop><|system|>\nRespond as a high quality storyteller.<|user|>\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-434a-51521-v1-uploader
Waiting for job on chaiml-reward-dpo-434a-51521-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-air-4-5-sft-62396-v3-uploader
Waiting for job on chaiml-glm-air-4-5-sft-62396-v3-uploader to finish
chaiml-reward-dpo-696c-43579-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-696c-43579-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-696c-chaiml-glm-air-4-5-sft-_62396_v1...
chaiml-reward-dpo-434a-51521-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-434a-51521-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-434a-chaiml-glm-air-4-5-sft-_62396_v1...
chaiml-glm-air-4-5-sft-62396-v3-uploader: Using quantization_mode: none
chaiml-glm-air-4-5-sft-62396-v3-uploader: Downloading snapshot of ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v1_e2...
chaiml-reward-dpo-696c-43579-v1-uploader: Downloaded in 71.624s
chaiml-reward-dpo-434a-51521-v1-uploader: Downloaded in 73.882s
chaiml-glm-air-4-5-sft-62396-v3-uploader: Downloaded in 73.682s
chaiml-reward-dpo-696c-43579-v1-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-696c-43579-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-696c-43579-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-reward-dpo-696c-43579-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-reward-dpo-696c-43579-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-reward-dpo-696c-43579-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-696c-43579-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-696c-43579-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-696c-43579-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-696c-43579-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-434a-51521-v1-uploader: Processed model ChaiML/reward-dpo-434a-chaiml-glm-air-4-5-sft-_62396_v1 in 151.052s
chaiml-reward-dpo-696c-43579-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-434a-51521-v1-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-696c-43579-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-434a-51521-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-696c-43579-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-434a-51521-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-reward-dpo-696c-43579-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-434a-51521-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-reward-dpo-696c-43579-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-434a-51521-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-reward-dpo-696c-43579-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-reward-dpo-434a-51521-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-696c-43579-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-reward-dpo-434a-51521-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-696c-43579-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-434a-51521-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-696c-43579-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default
chaiml-reward-dpo-434a-51521-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model.safetensors.index.json
chaiml-reward-dpo-434a-51521-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/.gitattributes
chaiml-reward-dpo-434a-51521-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/chat_template.jinja
chaiml-reward-dpo-434a-51521-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/config.json
chaiml-reward-dpo-434a-51521-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/special_tokens_map.json
chaiml-reward-dpo-434a-51521-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/tokenizer_config.json
chaiml-reward-dpo-434a-51521-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/README.md
chaiml-reward-dpo-434a-51521-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/args.json
chaiml-reward-dpo-434a-51521-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/tokenizer.json
chaiml-reward-dpo-434a-51521-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-434a-51521-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/README.md
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/tokenizer_config.json
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/special_tokens_map.json
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/.gitattributes
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/chat_template.jinja
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model.safetensors.index.json
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/config.json
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/args.json
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/tokenizer.json
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00043-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00004-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00010-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00033-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00014-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00028-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00016-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00007-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00035-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00001-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00003-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00019-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00026-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00025-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00017-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00011-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00018-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00013-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00021-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00020-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00030-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00023-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00037-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00034-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00022-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00012-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00002-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00038-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00024-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00015-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00039-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00008-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00031-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00032-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00036-of-00043.safetensors
chaiml-reward-dpo-696c-43579-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-696c-43579-v1/default/model-00009-of-00043.safetensors
Job chaiml-reward-dpo-696c-43579-v1-uploader completed after 237.28s with status: succeeded
Stopping job with name chaiml-reward-dpo-696c-43579-v1-uploader
Pipeline stage VLLMUploader completed in 239.43s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.51s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-reward-dpo-696c-43579-v1
Waiting for inference service chaiml-reward-dpo-696c-43579-v1 to be ready
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: Processed model ChaiML/glm_air_4_5_sft_20260228_thinking_plus_q_v1_e2 in 215.212s
chaiml-glm-air-4-5-sft-62396-v3-uploader: creating bucket guanaco-vllm-models
chaiml-glm-air-4-5-sft-62396-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-glm-air-4-5-sft-62396-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-glm-air-4-5-sft-62396-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-glm-air-4-5-sft-62396-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-62396-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-glm-air-4-5-sft-62396-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-62396-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-glm-air-4-5-sft-62396-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-glm-air-4-5-sft-62396-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-glm-air-4-5-sft-62396-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-glm-air-4-5-sft-62396-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-glm-air-4-5-sft-62396-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-glm-air-4-5-sft-62396-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-glm-air-4-5-sft-62396-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/README.md
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/chat_template.jinja
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/.gitattributes
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/special_tokens_map.json
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model.safetensors.index.json
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/config.json
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/args.json
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/tokenizer_config.json
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/tokenizer.json
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00022-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00018-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00008-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00038-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00033-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00011-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00019-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00002-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00023-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00003-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00026-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00025-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00009-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00020-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00039-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00016-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00001-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00007-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00010-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00037-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00035-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00015-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00004-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00034-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00030-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00017-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00031-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00024-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00032-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00021-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00013-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00012-of-00043.safetensors
chaiml-reward-dpo-434a-51521-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-434a-51521-v1/default/model-00028-of-00043.safetensors
Job chaiml-reward-dpo-434a-51521-v1-uploader completed after 291.22s with status: succeeded
Stopping job with name chaiml-reward-dpo-434a-51521-v1-uploader
Pipeline stage VLLMUploader completed in 293.35s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.19s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-reward-dpo-434a-51521-v1
Waiting for inference service chaiml-reward-dpo-434a-51521-v1 to be ready
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00043-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00020-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00039-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00027-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00037-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00019-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00024-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00042-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00040-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00034-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00009-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00033-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00004-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00013-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00011-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00014-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00022-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00002-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00001-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00008-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00015-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00031-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00028-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00041-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00029-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00012-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00010-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00038-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00007-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00035-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00032-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00036-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00017-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00023-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00005-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00025-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00018-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00016-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00003-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00006-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00030-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00026-of-00043.safetensors
chaiml-glm-air-4-5-sft-62396-v3-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-glm-air-4-5-sft-62396-v3/default/model-00021-of-00043.safetensors
Job chaiml-glm-air-4-5-sft-62396-v3-uploader completed after 360.22s with status: succeeded
Stopping job with name chaiml-glm-air-4-5-sft-62396-v3-uploader
Pipeline stage VLLMUploader completed in 362.53s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.08s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-glm-air-4-5-sft-62396-v3
Waiting for inference service chaiml-glm-air-4-5-sft-62396-v3 to be ready
Retrying (%r) after connection broken by '%r': %s
Inference service chaiml-reward-dpo-696c-43579-v1 ready after 607.7046468257904s
Pipeline stage VLLMDeployer completed in 609.99s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.515214443206787s
Received healthy response to inference request in 2.2602360248565674s
Received healthy response to inference request in 2.5295045375823975s
Received healthy response to inference request in 2.421980381011963s
Received healthy response to inference request in 2.385540246963501s
Received healthy response to inference request in 2.314683198928833s
Received healthy response to inference request in 2.1922976970672607s
Received healthy response to inference request in 7.2103376388549805s
Received healthy response to inference request in 2.547321081161499s
Received healthy response to inference request in 2.475945472717285s
Inference service chaiml-reward-dpo-434a-51521-v1 ready after 588.5199620723724s
Pipeline stage VLLMDeployer completed in 590.80s
Received healthy response to inference request in 2.2892069816589355s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.571643829345703s
Received healthy response to inference request in 2.5019612312316895s
Received healthy response to inference request in 2.484116315841675s
Received healthy response to inference request in 2.1990859508514404s
Received healthy response to inference request in 2.2595956325531006s
Received healthy response to inference request in 2.303164005279541s
Received healthy response to inference request in 2.165675640106201s
Received healthy response to inference request in 2.4983432292938232s
Received healthy response to inference request in 2.2095565795898438s
Received healthy response to inference request in 2.1201395988464355s
Received healthy response to inference request in 2.1553659439086914s
Received healthy response to inference request in 2.3466358184814453s
Received healthy response to inference request in 2.109239339828491s
Received healthy response to inference request in 2.2500650882720947s
Received healthy response to inference request in 2.3857460021972656s
Received healthy response to inference request in 2.3272998332977295s
Received healthy response to inference request in 2.26731014251709s
Received healthy response to inference request in 2.362067222595215s
Received healthy response to inference request in 2.146636962890625s
Received healthy response to inference request in 2.2085065841674805s
Received healthy response to inference request in 2.1689722537994385s
Received healthy response to inference request in 2.3316330909729004s
Received healthy response to inference request in 2.10587739944458s
Received healthy response to inference request in 2.2828831672668457s
Received healthy response to inference request in 2.1622753143310547s
Received healthy response to inference request in 2.2224318981170654s
Received healthy response to inference request in 2.120842218399048s
Received healthy response to inference request in 2.192577362060547s
Received healthy response to inference request in 2.1554291248321533s
Received healthy response to inference request in 2.2107391357421875s
Received healthy response to inference request in 2.1503994464874268s
Received healthy response to inference request in 2.2486751079559326s
Received healthy response to inference request in 2.263213872909546s
Received healthy response to inference request in 2.315058946609497s
Received healthy response to inference request in 2.202684164047241s
Received healthy response to inference request in 2.2086803913116455s
Received healthy response to inference request in 2.1374917030334473s
Received healthy response to inference request in 2.114274740219116s
30 requests
0 failed requests
5th percentile: 2.177655565738678
Inference service chaiml-glm-air-4-5-sft-62396-v3 ready after 570.969489812851s
10th percentile: 2.192549395561218
Received healthy response to inference request in 2.174990653991699s
Pipeline stage VLLMDeployer completed in 574.56s
20th percentile: 2.209381341934204
run pipeline stage %s
30th percentile: 2.2408021450042725
Running pipeline stage StressChecker
40th percentile: 2.2599798679351806
50th percentile: 2.3019450902938843
Received healthy response to inference request in 2.1513402462005615s
60th percentile: 2.329033136367798
70th percentile: 2.3691091299057008
Received healthy response to inference request in 2.317013740539551s
80th percentile: 2.477579641342163
90th percentile: 2.5497533559799193
95th percentile: 3.0906076669692966
Received healthy response to inference request in 2.18867826461792s
99th percentile: 6.138751912117008
mean time: 2.5151461362838745
Pipeline stage StressChecker completed in 102.85s
Received healthy response to inference request in 2.5463860034942627s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Received healthy response to inference request in 2.310991048812866s
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.456226348876953s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.13s
Received healthy response to inference request in 2.3529458045959473s
Shutdown handler de-registered
chaiml-reward-dpo-696c-_43579_v1 status is now deployed due to DeploymentManager action
Received healthy response to inference request in 2.1955764293670654s
Received healthy response to inference request in 2.2398488521575928s
Received healthy response to inference request in 2.0922696590423584s
Received healthy response to inference request in 2.3495359420776367s
Received healthy response to inference request in 2.1017355918884277s
Received healthy response to inference request in 2.147611618041992s
Received healthy response to inference request in 2.1132962703704834s
Received healthy response to inference request in 2.175462484359741s
Received healthy response to inference request in 2.608297348022461s
Received healthy response to inference request in 2.205763339996338s
Received healthy response to inference request in 2.2258048057556152s
Received healthy response to inference request in 2.0827813148498535s
30 requests
0 failed requests
5th percentile: 2.10739027261734
10th percentile: 2.119049572944641
20th percentile: 2.1448079109191895
Received healthy response to inference request in 2.485206127166748s
30th percentile: 2.151058006286621
40th percentile: 2.159536838531494
50th percentile: 2.17522656917572
60th percentile: 2.200525236129761
70th percentile: 2.2468583583831787
80th percentile: 2.3047294139862062
90th percentile: 2.356225824356079
Received healthy response to inference request in 2.0682287216186523s
95th percentile: 2.447674477100372
99th percentile: 2.5009120106697083
mean time: 2.2178042491277057
Pipeline stage StressChecker completed in 89.39s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.7593376636505127s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.06s
Shutdown handler de-registered
chaiml-reward-dpo-434a-_51521_v1 status is now deployed due to DeploymentManager action
chaiml-reward-dpo-434a-_51521_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-reward-dpo-434a-_51521_v1 status is now torndown due to DeploymentManager action