chaiml-reward-dpo-ed00-_15734

developer_uid: rirv938
submission_id: chaiml-reward-dpo-ed00-_15734_v1
model_name: chaiml-reward-dpo-ed00-_15734_v1
model_group: ChaiML/reward-dpo-ed00-c
status: inactive
timestamp: 2026-02-25T10:37:35+00:00
num_battles: 10173
num_wins: 5575
celo_rating: 1328.49
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/reward-dpo-ed00-chaiml-glm-air-4-5-sft-_92345_v1
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 9073971200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-reward-dpo-ed00-_15734_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/reward-dpo-ed00-chaiml-glm-air-4-5-sft-_92345_v1
model_size: 9B
ranking_group: single
us_pacific_date: 2026-02-25
win_ratio: 0.5480192666863265
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', 'You:', '<|im_start|>', '###', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '[gMASK]<sop><|system|>\nRespond as a high quality storyteller.<|user|>\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-ed00-15734-v1-uploader
Waiting for job on chaiml-reward-dpo-ed00-15734-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-b9ea-10046-v1-uploader
Waiting for job on chaiml-reward-dpo-b9ea-10046-v1-uploader to finish
chaiml-reward-dpo-ed00-15734-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-ed00-15734-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-ed00-chaiml-glm-air-4-5-sft-_92345_v1...
chaiml-reward-dpo-b9ea-10046-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-b9ea-10046-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-b9ea-chaiml-glm-air-4-5-sft-_92345_v1...
chaiml-reward-dpo-ed00-15734-v1-uploader: Downloaded in 74.644s
chaiml-reward-dpo-b9ea-10046-v1-uploader: Downloaded in 74.464s
chaiml-reward-dpo-ed00-15734-v1-uploader: Processed model ChaiML/reward-dpo-ed00-chaiml-glm-air-4-5-sft-_92345_v1 in 155.400s
chaiml-reward-dpo-ed00-15734-v1-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-reward-dpo-ed00-15734-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-ed00-15734-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-reward-dpo-ed00-15734-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-reward-dpo-ed00-15734-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-ed00-15734-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/.gitattributes
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/chat_template.jinja
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/args.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/tokenizer_config.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/special_tokens_map.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model.safetensors.index.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/README.md
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/config.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/tokenizer.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00043-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00028-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00003-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00026-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00018-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00002-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00034-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00033-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00007-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00011-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00020-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00009-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00037-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00013-of-00043.safetensors
Job chaiml-reward-dpo-ed00-15734-v1-uploader completed after 236.22s with status: succeeded
Stopping job with name chaiml-reward-dpo-ed00-15734-v1-uploader
Pipeline stage VLLMUploader completed in 237.65s
chaiml-reward-dpo-b9ea-10046-v1-uploader: Processed model ChaiML/reward-dpo-b9ea-chaiml-glm-air-4-5-sft-_92345_v1 in 214.729s
run pipeline stage %s
chaiml-reward-dpo-b9ea-10046-v1-uploader: creating bucket guanaco-vllm-models
Running pipeline stage VLLMTemplater
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
Pipeline stage VLLMTemplater completed in 0.46s
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
run pipeline stage %s
chaiml-reward-dpo-b9ea-10046-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
Running pipeline stage VLLMDeployer
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
Creating inference service chaiml-reward-dpo-ed00-15734-v1
chaiml-reward-dpo-b9ea-10046-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
Waiting for inference service chaiml-reward-dpo-ed00-15734-v1 to be ready
chaiml-reward-dpo-b9ea-10046-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-reward-dpo-b9ea-10046-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-b9ea-10046-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/.gitattributes
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/config.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/README.md
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/args.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/chat_template.jinja
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/special_tokens_map.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/tokenizer_config.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model.safetensors.index.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/tokenizer.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: b7fde52b-18da-99b9-a148-2b30c40efc3a, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e27127e2-cfe2-939b-a164-631be7fff1c8, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 0b96f46d-7f57-98ad-ac72-6247a11456c9, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 6cff346a-2f0e-9344-a7ed-798c7f4b9a02, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e547702d-3ee3-9fbf-ac0f-13067480a543, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e96343ab-909b-9a62-ab03-6c967ca905a3, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: d200d08d-947c-9604-9936-c8cd04ca84e6, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 040c46a6-40fb-9a40-bf91-1694339d9df0, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: debf450c-dc4c-9f51-86e6-706351c92823, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 05168135-c624-93b8-88c1-f8dc9a9d7e46, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 860efb57-f463-988e-8587-7d26faa163e7, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e3dce1b0-1320-93e3-ade4-6f78a810d8ce, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 0edcef82-491b-93f4-a58b-cb810e42be81, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: a1e7c0d5-453f-9077-86a9-e354d0a5d261, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 6321ee4e-bd4c-9354-b7c2-865adb8a0379, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: a65ea150-4f56-9492-bcf7-3445f12b941f, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e8ec5f20-dcc1-942f-b481-4cfdabf17070, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 5506fc4e-5bba-9fe3-825f-2104c622893b, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 663c390f-ffdb-9cb9-9b42-edb68634e607, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 864aae24-a645-95e2-82e0-39442844a6cd, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 51dc7376-b61f-9671-ab92-047a9defeb94, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 0d942be4-dcdc-923f-a7cd-4c60c429a98c, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 0c554657-bb55-993f-8e96-8d24e00cd97a, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 5b102980-d6dd-99b6-8e49-795cfeab90be, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 619fa56e-c535-90c2-ac1d-6f4d468cd0e9, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 4914655a-8925-9fa2-b0c5-da58f00b99ad, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 6acf4bf6-8a69-9d57-82b6-f3472bdbb5c4, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00004-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00020-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00036-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00019-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00038-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00011-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00033-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00028-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00010-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00003-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00001-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00007-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00039-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00009-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00014-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00025-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00021-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00015-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00030-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00032-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00035-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00012-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00037-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00008-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00013-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00022-of-00043.safetensors
Job chaiml-reward-dpo-b9ea-10046-v1-uploader completed after 624.28s with status: succeeded
Stopping job with name chaiml-reward-dpo-b9ea-10046-v1-uploader
Pipeline stage VLLMUploader completed in 626.85s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.48s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-reward-dpo-b9ea-10046-v1
Waiting for inference service chaiml-reward-dpo-b9ea-10046-v1 to be ready
Inference service chaiml-reward-dpo-ed00-15734-v1 ready after 559.4167757034302s
Pipeline stage VLLMDeployer completed in 562.06s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.908489465713501s
Received healthy response to inference request in 2.599097490310669s
Received healthy response to inference request in 2.2876408100128174s
Received healthy response to inference request in 2.4491288661956787s
Received healthy response to inference request in 2.4797348976135254s
Received healthy response to inference request in 2.3946452140808105s
Received healthy response to inference request in 2.273090362548828s
Received healthy response to inference request in 2.354440450668335s
Received healthy response to inference request in 2.5849533081054688s
Received healthy response to inference request in 2.174868583679199s
Received healthy response to inference request in 2.156348943710327s
Received healthy response to inference request in 2.171630620956421s
Received healthy response to inference request in 2.4739301204681396s
Received healthy response to inference request in 2.186124801635742s
Received healthy response to inference request in 5.2885050773620605s
Received healthy response to inference request in 2.225592613220215s
Received healthy response to inference request in 2.2573435306549072s
Received healthy response to inference request in 2.1706035137176514s
Received healthy response to inference request in 3.0322632789611816s
Received healthy response to inference request in 2.210523843765259s
Received healthy response to inference request in 2.500948429107666s
Received healthy response to inference request in 2.1842153072357178s
Received healthy response to inference request in 2.891193389892578s
Received healthy response to inference request in 2.282381534576416s
Received healthy response to inference request in 2.2001521587371826s
Received healthy response to inference request in 2.2389259338378906s
Received healthy response to inference request in 2.8449900150299072s
Received healthy response to inference request in 2.2657814025878906s
Received healthy response to inference request in 2.1975646018981934s
Received healthy response to inference request in 2.186889886856079s
30 requests
0 failed requests
5th percentile: 2.1710657119750976
10th percentile: 2.1745447874069215
20th percentile: 2.1867368698120115
30th percentile: 2.207412338256836
40th percentile: 2.2499764919281007
50th percentile: 2.277735948562622
60th percentile: 2.3705223560333253
70th percentile: 2.4756715536117553
80th percentile: 2.587782144546509
90th percentile: 2.8929229974746704
95th percentile: 2.976565062999725
99th percentile: 4.634194955825808
mean time: 2.4823999484380086
Pipeline stage StressChecker completed in 94.25s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.32s
Shutdown handler de-registered
chaiml-reward-dpo-ed00-_15734_v1 status is now deployed due to DeploymentManager action
chaiml-reward-dpo-ed00-_15734_v1 status is now inactive due to auto deactivation removed underperforming models