chaiml-reward-dpo-1044-_39258

developer_uid: rirv938
submission_id: chaiml-reward-dpo-1044-_39258_v1
model_name: chaiml-reward-dpo-1044-_39258_v1
model_group: ChaiML/reward-dpo-1044-c
status: torndown
timestamp: 2026-03-06T03:33:12+00:00
num_battles: 10644
num_wins: 5599
celo_rating: 1317.22
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/reward-dpo-1044-chaiml-glm-air-4-5-sft-_90753_v1
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 9073971200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-reward-dpo-1044-_39258_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/reward-dpo-1044-chaiml-glm-air-4-5-sft-_90753_v1
model_size: 9B
ranking_group: single
us_pacific_date: 2026-03-02
win_ratio: 0.5260240511086058
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', 'You:', '</s>', '###', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '[gMASK]<sop><|system|>\nRespond as a high quality storyteller.<|user|>\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-1044-39258-v1-uploader
Waiting for job on chaiml-reward-dpo-1044-39258-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-d9b1-68526-v1-uploader
Waiting for job on chaiml-reward-dpo-d9b1-68526-v1-uploader to finish
chaiml-reward-dpo-1044-39258-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-1044-39258-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-1044-chaiml-glm-air-4-5-sft-_90753_v1...
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-reward-dpo-b0ee-82897-v1-uploader
Waiting for job on chaiml-reward-dpo-b0ee-82897-v1-uploader to finish
chaiml-reward-dpo-d9b1-68526-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-d9b1-68526-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-d9b1-chaiml-glm-air-4-5-sft-_90753_v1...
chaiml-reward-dpo-b0ee-82897-v1-uploader: Using quantization_mode: none
chaiml-reward-dpo-b0ee-82897-v1-uploader: Downloading snapshot of ChaiML/reward-dpo-b0ee-chaiml-glm-air-4-5-sft-_90753_v1...
chaiml-reward-dpo-1044-39258-v1-uploader: Downloaded in 76.525s
chaiml-reward-dpo-d9b1-68526-v1-uploader: Downloaded in 89.698s
chaiml-reward-dpo-b0ee-82897-v1-uploader: Downloaded in 77.252s
chaiml-reward-dpo-1044-39258-v1-uploader: Processed model ChaiML/reward-dpo-1044-chaiml-glm-air-4-5-sft-_90753_v1 in 152.533s
chaiml-reward-dpo-b0ee-82897-v1-uploader: Processed model ChaiML/reward-dpo-b0ee-chaiml-glm-air-4-5-sft-_90753_v1 in 152.050s
chaiml-reward-dpo-b0ee-82897-v1-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-b0ee-82897-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b0ee-82897-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-reward-dpo-b0ee-82897-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-reward-dpo-b0ee-82897-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00043-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b0ee-82897-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-b0ee-82897-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b0ee-82897-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-b0ee-82897-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b0ee-82897-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-b0ee-82897-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-b0ee-82897-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-b0ee-82897-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-b0ee-82897-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-b0ee-82897-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00009-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00032-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00032-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00039-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00039-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00020-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/README.md
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00001-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/args.json
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00007-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00007-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/chat_template.jinja
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00022-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/.gitattributes
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00010-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model.safetensors.index.json
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00002-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00002-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/config.json
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00012-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/special_tokens_map.json
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00014-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/tokenizer_config.json
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00018-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/tokenizer.json
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00038-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00024-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00015-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00026-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00030-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00003-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00017-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00017-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00019-of-00043.safetensors
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00028-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00005-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00005-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: Processed model ChaiML/reward-dpo-d9b1-chaiml-glm-air-4-5-sft-_90753_v1 in 214.873s
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00004-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00023-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00023-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: creating bucket guanaco-vllm-models
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00034-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00001-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00001-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00004-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00004-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00036-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00022-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00031-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00021-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00025-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00012-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00012-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00021-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00021-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00042-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00013-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00014-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00014-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00037-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00037-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00025-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00025-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00033-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00033-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00029-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00029-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00028-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00028-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00011-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00011-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00010-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00010-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00006-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00006-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00016-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00027-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00027-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-reward-dpo-1044-39258-v1-uploader: cp /dev/shm/model_output/model-00036-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-1044-39258-v1/default/model-00036-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00018-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00018-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00024-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00024-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00041-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00041-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00019-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00019-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00016-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00016-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00035-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00035-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00040-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00040-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00031-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00031-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00030-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00030-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: cp /dev/shm/model_output/model-00043-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d9b1-68526-v1/default/model-00043-of-00043.safetensors
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00009-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00009-of-00043.safetensors
Job chaiml-reward-dpo-1044-39258-v1-uploader completed after 274.2s with status: succeeded
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00015-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00015-of-00043.safetensors
Stopping job with name chaiml-reward-dpo-1044-39258-v1-uploader
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00020-of-00043.safetensors
Pipeline stage VLLMUploader completed in 281.29s
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00038-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00038-of-00043.safetensors
run pipeline stage %s
chaiml-reward-dpo-b0ee-82897-v1-uploader: cp /dev/shm/model_output/model-00003-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-b0ee-82897-v1/default/model-00003-of-00043.safetensors
Running pipeline stage VLLMTemplater
chaiml-reward-dpo-d9b1-68526-v1-uploader: cp /dev/shm/model_output/model-00020-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d9b1-68526-v1/default/model-00020-of-00043.safetensors
Pipeline stage VLLMTemplater completed in 3.16s
chaiml-reward-dpo-d9b1-68526-v1-uploader: cp /dev/shm/model_output/model-00022-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d9b1-68526-v1/default/model-00022-of-00043.safetensors
run pipeline stage %s
chaiml-reward-dpo-d9b1-68526-v1-uploader: cp /dev/shm/model_output/model-00026-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d9b1-68526-v1/default/model-00026-of-00043.safetensors
Running pipeline stage VLLMDeployer
chaiml-reward-dpo-d9b1-68526-v1-uploader: cp /dev/shm/model_output/model-00013-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d9b1-68526-v1/default/model-00013-of-00043.safetensors
Creating inference service chaiml-reward-dpo-1044-39258-v1
chaiml-reward-dpo-d9b1-68526-v1-uploader: cp /dev/shm/model_output/model-00034-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d9b1-68526-v1/default/model-00034-of-00043.safetensors
chaiml-reward-dpo-d9b1-68526-v1-uploader: cp /dev/shm/model_output/model-00042-of-00043.safetensors s3://guanaco-vllm-models/chaiml-reward-dpo-d9b1-68526-v1/default/model-00042-of-00043.safetensors
Waiting for inference service chaiml-reward-dpo-1044-39258-v1 to be ready
Job chaiml-reward-dpo-b0ee-82897-v1-uploader completed after 272.84s with status: succeeded
Stopping job with name chaiml-reward-dpo-b0ee-82897-v1-uploader
Pipeline stage VLLMUploader completed in 280.33s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.57s
Job chaiml-reward-dpo-d9b1-68526-v1-uploader completed after 295.28s with status: succeeded
run pipeline stage %s
Stopping job with name chaiml-reward-dpo-d9b1-68526-v1-uploader
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploader completed in 300.15s
Creating inference service chaiml-reward-dpo-b0ee-82897-v1
run pipeline stage %s
Waiting for inference service chaiml-reward-dpo-b0ee-82897-v1 to be ready
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.81s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-reward-dpo-d9b1-68526-v1
Waiting for inference service chaiml-reward-dpo-d9b1-68526-v1 to be ready
Inference service chaiml-reward-dpo-1044-39258-v1 ready after 582.1946132183075s
Pipeline stage VLLMDeployer completed in 589.60s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service chaiml-reward-dpo-b0ee-82897-v1 ready after 570.4972398281097s
Pipeline stage VLLMDeployer completed in 578.07s
run pipeline stage %s
Received healthy response to inference request in 6.508826732635498s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.170912742614746s
Received healthy response to inference request in 4.745554208755493s
Received healthy response to inference request in 4.230443954467773s
Received healthy response to inference request in 4.857105255126953s
Received healthy response to inference request in 5.038939952850342s
Received healthy response to inference request in 3.3851490020751953s
Received healthy response to inference request in 3.1830456256866455s
Received healthy response to inference request in 4.083965301513672s
Received healthy response to inference request in 6.014947891235352s
Received healthy response to inference request in 3.533442497253418s
Received healthy response to inference request in 2.8944873809814453s
Received healthy response to inference request in 3.352712869644165s
Received healthy response to inference request in 9.138283252716064s
Received healthy response to inference request in 7.425544738769531s
Received healthy response to inference request in 3.566768169403076s
Received healthy response to inference request in 3.6807243824005127s
Received healthy response to inference request in 4.67350172996521s
Received healthy response to inference request in 3.1271443367004395s
Received healthy response to inference request in 4.728521823883057s
Received healthy response to inference request in 3.836097240447998s
Received healthy response to inference request in 3.1165616512298584s
Received healthy response to inference request in 2.7975921630859375s
Received healthy response to inference request in 4.659898042678833s
Received healthy response to inference request in 3.96530818939209s
Received healthy response to inference request in 3.32319712638855s
Received healthy response to inference request in 3.1055357456207275s
Received healthy response to inference request in 4.636388063430786s
Received healthy response to inference request in 3.783607006072998s
Received healthy response to inference request in 2.802870750427246s
Received healthy response to inference request in 3.0747034549713135s
Received healthy response to inference request in 6.986799955368042s
Received healthy response to inference request in 13.652459144592285s
Received healthy response to inference request in 15.262922763824463s
Received healthy response to inference request in 10.893314123153687s
Received healthy response to inference request in 5.451234340667725s
Received healthy response to inference request in 3.621187210083008s
Received healthy response to inference request in 4.79976487159729s
Received healthy response to inference request in 4.26019811630249s
Received healthy response to inference request in 4.6345274448394775s
Received healthy response to inference request in 4.09852933883667s
Received healthy response to inference request in 2.956503391265869s
Received healthy response to inference request in 3.2062251567840576s
Received healthy response to inference request in 3.7041525840759277s
Received healthy response to inference request in 4.289891004562378s
Received healthy response to inference request in 3.565707206726074s
Received healthy response to inference request in 4.717375040054321s
Received healthy response to inference request in 3.30077862739563s
Received healthy response to inference request in 3.002894639968872s
Received healthy response to inference request in 5.362760782241821s
Received healthy response to inference request in 4.291159629821777s
Received healthy response to inference request in 3.567808151245117s
Received healthy response to inference request in 3.0580384731292725s
Received healthy response to inference request in 5.2096381187438965s
Received healthy response to inference request in 4.668217182159424s
Received healthy response to inference request in 2.915046453475952s
Received healthy response to inference request in 8.438023567199707s
Received healthy response to inference request in 3.549656629562378s
Received healthy response to inference request in 4.9222412109375s
Received healthy response to inference request in 5.902876853942871s
30 requests
30 requests
0 failed requests
0 failed requests
5th percentile: 2.9944208025932313
5th percentile: 2.9337020754814147
10th percentile: 3.1654776334762573
10th percentile: 2.9982555150985717
20th percentile: 3.2818679332733156
20th percentile: 3.0993692874908447
30th percentile: 3.5664498805999756
30th percentile: 3.3754181623458863
40th percentile: 4.266112184524536
40th percentile: 3.5925749778747558
50th percentile: 4.64814305305481
50th percentile: 3.743879795074463
60th percentile: 4.75701904296875
60th percentile: 4.012771034240722
70th percentile: 5.090149402618407
70th percentile: 4.269486570358277
80th percentile: 5.563977050781252
80th percentile: 4.723010873794555
90th percentile: 7.1319223165512105
90th percentile: 6.05514364242554
95th percentile: 8.823166394233702
95th percentile: 9.332817900180807
99th percentile: 13.486777305603033
99th percentile: 12.852307088375094
mean time: 5.0228639364242555
mean time: 4.468193173408508
Pipeline stage StressChecker completed in 255.21s
Pipeline stage StressChecker completed in 247.06s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 10.38s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 10.75s
Shutdown handler de-registered
Shutdown handler de-registered
chaiml-reward-dpo-1044-_39258_v1 status is now deployed due to DeploymentManager action
chaiml-reward-dpo-b0ee-_82897_v1 status is now deployed due to DeploymentManager action
chaiml-reward-dpo-1044-_39258_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-reward-dpo-1044-_39258_v1 status is now torndown due to DeploymentManager action