developer_uid: rirv938
submission_id: chaiml-reward-dpo-b9ea-_10046_v1
model_name: chaiml-reward-dpo-b9ea-_10046_v1
model_group: ChaiML/reward-dpo-b9ea-c
status: inactive
timestamp: 2026-02-25T10:47:26+00:00
num_battles: 10277
num_wins: 5705
celo_rating: 1337.18
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/reward-dpo-b9ea-chaiml-glm-air-4-5-sft-_92345_v1
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 9073971200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-reward-dpo-b9ea-_10046_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/reward-dpo-b9ea-chaiml-glm-air-4-5-sft-_92345_v1
model_size: 9B
ranking_group: single
us_pacific_date: 2026-02-25
win_ratio: 0.5551230903960299
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '<|im_start|>', '</s>', '###', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '[gMASK]<sop><|system|>\nRespond as a high quality storyteller.<|user|>\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-b9ea-10046-v1-uploader
Waiting for job on chaiml-reward-dpo-b9ea-10046-v1-uploader to finish
chaiml-reward-dpo-ed00-15734-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-ed00-15734-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-ed00-chaiml-glm-air-4-5-sft-_92345_v1...
chaiml-reward-dpo-b9ea-10046-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-b9ea-10046-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-b9ea-chaiml-glm-air-4-5-sft-_92345_v1...
chaiml-reward-dpo-ed00-15734-v1-uploader: Downloaded in 74.644s
chaiml-reward-dpo-b9ea-10046-v1-uploader: Downloaded in 74.464s
chaiml-reward-dpo-ed00-15734-v1-uploader: Processed model ChaiML/reward-dpo-ed00-chaiml-glm-air-4-5-sft-_92345_v1 in 155.400s
chaiml-reward-dpo-ed00-15734-v1-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-reward-dpo-ed00-15734-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-ed00-15734-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-ed00-15734-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-ed00-15734-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-reward-dpo-ed00-15734-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-reward-dpo-ed00-15734-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-ed00-15734-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/.gitattributes
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/chat_template.jinja
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/args.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/tokenizer_config.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/special_tokens_map.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model.safetensors.index.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/README.md
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/config.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/tokenizer.json
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00043-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00028-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00003-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00026-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00018-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00002-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00034-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00033-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00007-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00011-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00020-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00009-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00037-of-00043.safetensors
chaiml-reward-dpo-ed00-15734-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-ed00-15734-v1/default/model-00013-of-00043.safetensors
Job chaiml-reward-dpo-ed00-15734-v1-uploader completed after 236.22s with status: succeeded
Stopping job with name chaiml-reward-dpo-ed00-15734-v1-uploader
Pipeline stage VLLMUploader completed in 237.65s
chaiml-reward-dpo-b9ea-10046-v1-uploader: Processed model ChaiML/reward-dpo-b9ea-chaiml-glm-air-4-5-sft-_92345_v1 in 214.729s
run pipeline stage %s
chaiml-reward-dpo-b9ea-10046-v1-uploader: creating bucket guanaco-vllm-models
Running pipeline stage VLLMTemplater
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
Pipeline stage VLLMTemplater completed in 0.46s
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
run pipeline stage %s
chaiml-reward-dpo-b9ea-10046-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
Running pipeline stage VLLMDeployer
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
Creating inference service chaiml-reward-dpo-ed00-15734-v1
chaiml-reward-dpo-b9ea-10046-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
Waiting for inference service chaiml-reward-dpo-ed00-15734-v1 to be ready
chaiml-reward-dpo-b9ea-10046-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-b9ea-10046-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-reward-dpo-b9ea-10046-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-reward-dpo-b9ea-10046-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-b9ea-10046-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/.gitattributes
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/config.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/README.md
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/args.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/chat_template.jinja
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/special_tokens_map.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/tokenizer_config.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model.safetensors.index.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/tokenizer.json
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: b7fde52b-18da-99b9-a148-2b30c40efc3a, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e27127e2-cfe2-939b-a164-631be7fff1c8, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 0b96f46d-7f57-98ad-ac72-6247a11456c9, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 6cff346a-2f0e-9344-a7ed-798c7f4b9a02, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e547702d-3ee3-9fbf-ac0f-13067480a543, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e96343ab-909b-9a62-ab03-6c967ca905a3, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: d200d08d-947c-9604-9936-c8cd04ca84e6, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 040c46a6-40fb-9a40-bf91-1694339d9df0, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: debf450c-dc4c-9f51-86e6-706351c92823, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 05168135-c624-93b8-88c1-f8dc9a9d7e46, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 860efb57-f463-988e-8587-7d26faa163e7, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e3dce1b0-1320-93e3-ade4-6f78a810d8ce, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 0edcef82-491b-93f4-a58b-cb810e42be81, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: a1e7c0d5-453f-9077-86a9-e354d0a5d261, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 6321ee4e-bd4c-9354-b7c2-865adb8a0379, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: a65ea150-4f56-9492-bcf7-3445f12b941f, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: e8ec5f20-dcc1-942f-b481-4cfdabf17070, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 5506fc4e-5bba-9fe3-825f-2104c622893b, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 663c390f-ffdb-9cb9-9b42-edb68634e607, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 864aae24-a645-95e2-82e0-39442844a6cd, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 51dc7376-b61f-9671-ab92-047a9defeb94, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 0d942be4-dcdc-923f-a7cd-4c60c429a98c, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 0c554657-bb55-993f-8e96-8d24e00cd97a, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 5b102980-d6dd-99b6-8e49-795cfeab90be, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 619fa56e-c535-90c2-ac1d-6f4d468cd0e9, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 4914655a-8925-9fa2-b0c5-da58f00b99ad, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 500, request id: 6acf4bf6-8a69-9d57-82b6-f3472bdbb5c4, host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00004-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00020-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00036-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00019-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00038-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00011-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00033-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00028-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00010-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00003-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00001-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00007-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00039-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00009-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00014-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00025-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00021-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00015-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00030-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00032-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00035-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00012-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00037-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00008-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00008-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: DEBUG retryable error: GatewayTimeout: Gateway Timeout
chaiml-reward-dpo-b9ea-10046-v1-uploader: 	status code: 504, request id: , host id: 
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00013-of-00043.safetensors
chaiml-reward-dpo-b9ea-10046-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b9ea-10046-v1/default/model-00022-of-00043.safetensors
Job chaiml-reward-dpo-b9ea-10046-v1-uploader completed after 624.28s with status: succeeded
Stopping job with name chaiml-reward-dpo-b9ea-10046-v1-uploader
Pipeline stage VLLMUploader completed in 626.85s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.48s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-reward-dpo-b9ea-10046-v1
Waiting for inference service chaiml-reward-dpo-b9ea-10046-v1 to be ready
Inference service chaiml-reward-dpo-ed00-15734-v1 ready after 559.4167757034302s
Pipeline stage VLLMDeployer completed in 562.06s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.908489465713501s
Received healthy response to inference request in 2.599097490310669s
Received healthy response to inference request in 2.2876408100128174s
Received healthy response to inference request in 2.4491288661956787s
Received healthy response to inference request in 2.4797348976135254s
Received healthy response to inference request in 2.3946452140808105s
Received healthy response to inference request in 2.273090362548828s
Received healthy response to inference request in 2.354440450668335s
Received healthy response to inference request in 2.5849533081054688s
Received healthy response to inference request in 2.174868583679199s
Received healthy response to inference request in 2.156348943710327s
Received healthy response to inference request in 2.171630620956421s
Received healthy response to inference request in 2.4739301204681396s
Received healthy response to inference request in 2.186124801635742s
Received healthy response to inference request in 5.2885050773620605s
Received healthy response to inference request in 2.225592613220215s
Received healthy response to inference request in 2.2573435306549072s
Received healthy response to inference request in 2.1706035137176514s
Received healthy response to inference request in 3.0322632789611816s
Received healthy response to inference request in 2.210523843765259s
Received healthy response to inference request in 2.500948429107666s
Received healthy response to inference request in 2.1842153072357178s
Received healthy response to inference request in 2.891193389892578s
Received healthy response to inference request in 2.282381534576416s
Received healthy response to inference request in 2.2001521587371826s
Received healthy response to inference request in 2.2389259338378906s
Received healthy response to inference request in 2.8449900150299072s
Received healthy response to inference request in 2.2657814025878906s
Received healthy response to inference request in 2.1975646018981934s
Received healthy response to inference request in 2.186889886856079s
30 requests
0 failed requests
5th percentile: 2.1710657119750976
10th percentile: 2.1745447874069215
20th percentile: 2.1867368698120115
30th percentile: 2.207412338256836
40th percentile: 2.2499764919281007
50th percentile: 2.277735948562622
60th percentile: 2.3705223560333253
70th percentile: 2.4756715536117553
80th percentile: 2.587782144546509
90th percentile: 2.8929229974746704
95th percentile: 2.976565062999725
99th percentile: 4.634194955825808
mean time: 2.4823999484380086
Pipeline stage StressChecker completed in 94.25s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.32s
Shutdown handler de-registered
chaiml-reward-dpo-ed00-_15734_v1 status is now deployed due to DeploymentManager action
Inference service chaiml-reward-dpo-b9ea-10046-v1 ready after 567.8536894321442s
Pipeline stage VLLMDeployer completed in 569.46s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.4990642070770264s
Received healthy response to inference request in 2.1128385066986084s
Received healthy response to inference request in 2.2097485065460205s
Received healthy response to inference request in 2.1998300552368164s
Received healthy response to inference request in 2.2297134399414062s
Received healthy response to inference request in 2.2943813800811768s
Received healthy response to inference request in 2.087824821472168s
Received healthy response to inference request in 2.0964090824127197s
Received healthy response to inference request in 2.274738311767578s
Received healthy response to inference request in 2.235442876815796s
Received healthy response to inference request in 2.2116899490356445s
Received healthy response to inference request in 2.1423261165618896s
Received healthy response to inference request in 2.104581356048584s
Received healthy response to inference request in 2.0852911472320557s
Received healthy response to inference request in 2.2692363262176514s
Received healthy response to inference request in 2.1242263317108154s
Received healthy response to inference request in 2.2213830947875977s
Received healthy response to inference request in 2.2485361099243164s
Received healthy response to inference request in 2.2124664783477783s
Received healthy response to inference request in 2.1746556758880615s
Received healthy response to inference request in 2.2585575580596924s
Received healthy response to inference request in 2.211536169052124s
Received healthy response to inference request in 2.1457173824310303s
Received healthy response to inference request in 2.160130023956299s
Received healthy response to inference request in 2.1560800075531006s
Received healthy response to inference request in 2.1059346199035645s
Received healthy response to inference request in 2.15787410736084s
Received healthy response to inference request in 2.781306505203247s
Received healthy response to inference request in 2.1189088821411133s
Received healthy response to inference request in 2.1297898292541504s
30 requests
0 failed requests
5th percentile: 2.0916877388954163
10th percentile: 2.1037641286849977
20th percentile: 2.1176948070526125
30th percentile: 2.1385652303695677
40th percentile: 2.157156467437744
50th percentile: 2.187242865562439
60th percentile: 2.2115976810455322
70th percentile: 2.2238821983337402
80th percentile: 2.2505403995513915
90th percentile: 2.276702618598938
95th percentile: 2.4069569349288935
99th percentile: 2.6994562387466434
mean time: 2.2086739619572957
Pipeline stage StressChecker completed in 75.67s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.50s
Shutdown handler de-registered
chaiml-reward-dpo-b9ea-_10046_v1 status is now deployed due to DeploymentManager action
chaiml-reward-dpo-b9ea-_10046_v1 status is now inactive due to auto deactivation removed underperforming models