developer_uid: rirv938
submission_id: chaiml-235b-sft-prod-rm_88661_v1
model_name: chaiml-235b-sft-prod-rm_88661_v1
model_group: ChaiML/235b_sft_prod_rm_
status: torndown
timestamp: 2026-02-22T21:27:05+00:00
num_battles: 11479
num_wins: 6234
celo_rating: 1332.43
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-235b-sft-prod-rm_88661_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-16
win_ratio: 0.5430786653889712
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '<|im_end|>', '<|im_start|>', '###', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-prod-rm-88661-v1-uploader
Waiting for job on chaiml-235b-sft-prod-rm-88661-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-prod-rm-18463-v1-uploader
Waiting for job on chaiml-235b-sft-prod-rm-18463-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-prod-rm-38783-v1-uploader
Waiting for job on chaiml-235b-sft-prod-rm-38783-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-prod-rm-63215-v1-uploader
Waiting for job on chaiml-235b-sft-prod-rm-63215-v1-uploader to finish
chaiml-235b-sft-prod-rm-5716-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-5716-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e5-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-5716-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e5-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-5716-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-5716-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e5-W4A16...
chaiml-235b-sft-prod-rm-88661-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-88661-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-88661-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-88661-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-88661-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16...
chaiml-235b-sft-prod-rm-18463-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e3-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-63215-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-18463-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e3-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-63215-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-18463-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-63215-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-18463-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e3-W4A16...
chaiml-235b-sft-prod-rm-63215-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-63215-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16...
chaiml-235b-sft-prod-rm-38783-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-38783-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-38783-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-38783-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-38783-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16...
chaiml-235b-sft-prod-rm-5716-v1-uploader: Downloaded in 52.690s
chaiml-235b-sft-prod-rm-5716-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e5-W4A16 in 53.337s
chaiml-235b-sft-prod-rm-18463-v1-uploader: Downloaded in 51.253s
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/.gitattributes
chaiml-235b-sft-prod-rm-18463-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e3-W4A16 in 51.899s
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/chat_template.jinja
chaiml-235b-sft-prod-rm-18463-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/generation_config.json
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-18463-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/added_tokens.json
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-18463-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/quantization_config.json
chaiml-235b-sft-prod-rm-38783-v1-uploader: Downloaded in 52.351s
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/config.json
chaiml-235b-sft-prod-rm-38783-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 in 53.016s
chaiml-235b-sft-prod-rm-18463-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/merges.txt
chaiml-235b-sft-prod-rm-88661-v1-uploader: Downloaded in 53.115s
chaiml-235b-sft-prod-rm-38783-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/tokenizer.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 in 53.672s
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-38783-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/vocab.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-88661-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-38783-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-88661-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-38783-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-88661-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-88661-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-63215-v1-uploader: Downloaded in 53.310s
chaiml-235b-sft-prod-rm-18463-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-63215-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 in 53.868s
chaiml-235b-sft-prod-rm-18463-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default
chaiml-235b-sft-prod-rm-88661-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-38783-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-63215-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/.gitattributes
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/added_tokens.json
chaiml-235b-sft-prod-rm-88661-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-38783-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-63215-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/chat_template.jinja
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-88661-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-63215-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-63215-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-88661-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default
chaiml-235b-sft-prod-rm-63215-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/added_tokens.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/chat_template.jinja
chaiml-235b-sft-prod-rm-63215-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/.gitattributes
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/generation_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/quantization_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/merges.txt
chaiml-235b-sft-prod-rm-63215-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/vocab.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/added_tokens.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/tokenizer.json
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/tokenizer.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/generation_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/.gitattributes
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/chat_template.jinja
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/quantization_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/merges.txt
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00006-of-00027.safetensors?partNumber=7&uploadId=2~AaVb3XrLHbm6wYTmWy_bfZy9EBj6179": write tcp 10.0.50.49:60586->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00021-of-00027.safetensors?partNumber=6&uploadId=2~R7Uc_sIHOwPAsejkX7hi47o35JLmR76": write tcp 10.0.50.49:60560->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/tokenizer.json
chaiml-235b-sft-prod-rm-38783-v1-uploader: ERROR "cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00021-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~R7Uc_sIHOwPAsejkX7hi47o35JLmR76 caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000092f976f82401fcd-0069935327-156987ee57-default, host id:
chaiml-235b-sft-prod-rm-38783-v1-uploader: ERROR "cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00006-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~AaVb3XrLHbm6wYTmWy_bfZy9EBj6179 caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000ae749a2a989b0feb-0069935327-15656b5c61-default, host id:
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00002-of-00027.safetensors?partNumber=1&uploadId=2~KEjbczxMbDbiE4YPF-Z90M1A7d4SCJy": write tcp 10.0.50.49:60550->216.153.53.63:443: write: connection timed out
chaiml-235b-sft-prod-rm-38783-v1-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00002-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~KEjbczxMbDbiE4YPF-Z90M1A7d4SCJy caused by: SignatureDoesNotMatch:  status code: 403, request id: tx000007728fab36a2414db-00699353c3-15656b5c61-default, host id:
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00022-of-00027.safetensors?partNumber=84&uploadId=2~yQRTvmu_rZbCpGDqgUl1YMlm1iOD-h8": write tcp 10.0.50.49:56256->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-prod-rm-38783-v1-uploader: ERROR "cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00022-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~yQRTvmu_rZbCpGDqgUl1YMlm1iOD-h8 caused by: SignatureDoesNotMatch:  status code: 403, request id: tx000004cce0a1fa2be60fb-006993557e-156987ee57-default, host id:
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-5716-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00015-of-00027.safetensors?partNumber=76&uploadId=2~6_pRoXmSyvl3oI9udEQzr94Zu_6hDP3": write tcp 10.0.39.88:50612->216.153.53.63:443: use of closed network connection
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/.gitattributes": object size matches
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/added_tokens.json": object size matches
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/chat_template.jinja": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/.gitattributes": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/added_tokens.json": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/generation_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/chat_template.jinja": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/merges.txt": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/config.json": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00001-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/generation_config.json": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00003-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/merges.txt": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00004-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00001-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00005-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00002-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00007-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00003-of-00027.safetensors": object size matches
Job chaiml-235b-sft-prod-rm-18463-v1-uploader completed after 937.54s with status: succeeded
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00008-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00004-of-00027.safetensors": object size matches
Job chaiml-235b-sft-prod-rm-63215-v1-uploader completed after 927.59s with status: succeeded
Stopping job with name chaiml-235b-sft-prod-rm-18463-v1-uploader
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00009-of-00027.safetensors": object size matches
Job chaiml-235b-sft-prod-rm-88661-v1-uploader completed after 944.08s with status: succeeded
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00005-of-00027.safetensors": object size matches
Stopping job with name chaiml-235b-sft-prod-rm-63215-v1-uploader
Pipeline stage VLLMUploader completed in 943.07s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00010-of-00027.safetensors": object size matches
Stopping job with name chaiml-235b-sft-prod-rm-88661-v1-uploader
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00006-of-00027.safetensors": object size matches
Pipeline stage VLLMUploader completed in 933.65s
run pipeline stage %s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00011-of-00027.safetensors": object size matches
Pipeline stage VLLMUploader completed in 949.74s
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00007-of-00027.safetensors": object size matches
run pipeline stage %s
Running pipeline stage VLLMTemplater
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00012-of-00027.safetensors": object size matches
run pipeline stage %s
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00008-of-00027.safetensors": object size matches
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.90s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00013-of-00027.safetensors": object size matches
Running pipeline stage VLLMTemplater
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00009-of-00027.safetensors": object size matches
run pipeline stage %s
Pipeline stage VLLMTemplater completed in 1.70s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00014-of-00027.safetensors": object size matches
Pipeline stage VLLMTemplater completed in 2.01s
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00010-of-00027.safetensors": object size matches
Running pipeline stage VLLMDeployer
run pipeline stage %s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00015-of-00027.safetensors": object size matches
run pipeline stage %s
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00011-of-00027.safetensors": object size matches
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-prod-rm-18463-v1
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00016-of-00027.safetensors": object size matches
Running pipeline stage VLLMDeployer
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00012-of-00027.safetensors": object size matches
Creating inference service chaiml-235b-sft-prod-rm-63215-v1
Waiting for inference service chaiml-235b-sft-prod-rm-18463-v1 to be ready
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00017-of-00027.safetensors": object size matches
Creating inference service chaiml-235b-sft-prod-rm-88661-v1
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00013-of-00027.safetensors": object size matches
Waiting for inference service chaiml-235b-sft-prod-rm-63215-v1 to be ready
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00018-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00014-of-00027.safetensors": object size matches
Waiting for inference service chaiml-235b-sft-prod-rm-88661-v1 to be ready
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00019-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00016-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00020-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00017-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00023-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00018-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00024-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00019-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00025-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00020-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00026-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00021-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00027-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00022-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model.safetensors.index.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00023-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/quantization_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00024-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/special_tokens_map.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00025-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/tokenizer.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00026-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/tokenizer_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00027-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/vocab.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model.safetensors.index.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/quantization_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/special_tokens_map.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/tokenizer.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/tokenizer_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/vocab.json": object size matches
Job chaiml-235b-sft-prod-rm-38783-v1-uploader completed after 977.62s with status: succeeded
Stopping job with name chaiml-235b-sft-prod-rm-38783-v1-uploader
Pipeline stage VLLMUploader completed in 981.40s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.50s
Job chaiml-235b-sft-prod-rm-5716-v1-uploader completed after 999.97s with status: succeeded
run pipeline stage %s
Stopping job with name chaiml-235b-sft-prod-rm-5716-v1-uploader
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploader completed in 1004.30s
Creating inference service chaiml-235b-sft-prod-rm-38783-v1
run pipeline stage %s
Waiting for inference service chaiml-235b-sft-prod-rm-38783-v1 to be ready
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.54s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-prod-rm-5716-v1
Waiting for inference service chaiml-235b-sft-prod-rm-5716-v1 to be ready
Inference service chaiml-235b-sft-prod-rm-63215-v1 ready after 780.560982465744s
Pipeline stage VLLMDeployer completed in 786.40s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.9106545448303223s
Received healthy response to inference request in 2.4552035331726074s
Received healthy response to inference request in 2.692019462585449s
Received healthy response to inference request in 2.474421739578247s
Received healthy response to inference request in 2.7645368576049805s
Received healthy response to inference request in 2.5120503902435303s
Received healthy response to inference request in 2.63553524017334s
Received healthy response to inference request in 2.5533299446105957s
Received healthy response to inference request in 2.368685722351074s
Received healthy response to inference request in 2.4482309818267822s
Received healthy response to inference request in 2.7424418926239014s
Received healthy response to inference request in 2.581507921218872s
Received healthy response to inference request in 2.382631778717041s
Received healthy response to inference request in 2.3774311542510986s
Received healthy response to inference request in 2.942600965499878s
Received healthy response to inference request in 2.3730084896087646s
Received healthy response to inference request in 2.4892466068267822s
Received healthy response to inference request in 2.5626676082611084s
Received healthy response to inference request in 2.5661089420318604s
Received healthy response to inference request in 2.7901852130889893s
Received healthy response to inference request in 2.701385498046875s
Received healthy response to inference request in 2.471928834915161s
Received healthy response to inference request in 2.7403275966644287s
Received healthy response to inference request in 2.500352144241333s
Received healthy response to inference request in 2.493736505508423s
Received healthy response to inference request in 2.47042179107666s
Received healthy response to inference request in 2.5407485961914062s
Received healthy response to inference request in 2.4436745643615723s
Received healthy response to inference request in 2.6298956871032715s
Received healthy response to inference request in 2.8829574584960938s
30 requests
0 failed requests
5th percentile: 2.374998688697815
10th percentile: 2.3821117162704466
20th percentile: 2.4538090229034424
30th percentile: 2.473673868179321
40th percentile: 2.497705888748169
50th percentile: 2.547039270401001
60th percentile: 2.572268533706665
70th percentile: 2.6524805068969726
80th percentile: 2.740750455856323
90th percentile: 2.7994624376296997
95th percentile: 2.9157613873481747
99th percentile: 3.6299190068244944
mean time: 2.6165975888570148
Pipeline stage StressChecker completed in 137.36s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 6.10s
Shutdown handler de-registered
chaiml-235b-sft-prod-rm_63215_v1 status is now deployed due to DeploymentManager action
Inference service chaiml-235b-sft-prod-rm-18463-v1 ready after 992.4880702495575s
Pipeline stage VLLMDeployer completed in 998.30s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.6696324348449707s
Received healthy response to inference request in 2.814812660217285s
Inference service chaiml-235b-sft-prod-rm-88661-v1 ready after 1002.9109418392181s
Pipeline stage VLLMDeployer completed in 1008.30s
run pipeline stage %s
Received healthy response to inference request in 3.384521961212158s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.484959602355957s
Received healthy response to inference request in 2.4508891105651855s
Received healthy response to inference request in 2.519897699356079s
Received healthy response to inference request in 2.791884183883667s
Inference service chaiml-235b-sft-prod-rm-38783-v1 ready after 972.816748380661s
Pipeline stage VLLMDeployer completed in 978.40s
run pipeline stage %s
Received healthy response to inference request in 2.7720823287963867s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.7299442291259766s
Received healthy response to inference request in 2.635684013366699s
Received healthy response to inference request in 2.4506049156188965s
Received healthy response to inference request in 2.661996841430664s
Received healthy response to inference request in 2.605320453643799s
Received healthy response to inference request in 2.672523021697998s
Received healthy response to inference request in 2.508715867996216s
Received healthy response to inference request in 2.5433101654052734s
Received healthy response to inference request in 2.9567081928253174s
Received healthy response to inference request in 2.265794038772583s
Received healthy response to inference request in 2.4464967250823975s
Received healthy response to inference request in 2.4758853912353516s
Received healthy response to inference request in 2.4864423274993896s
Received healthy response to inference request in 2.3362131118774414s
Received healthy response to inference request in 2.8299663066864014s
Received healthy response to inference request in 3.1753876209259033s
Received healthy response to inference request in 2.6456918716430664s
Received healthy response to inference request in 2.7112505435943604s
Received healthy response to inference request in 2.4090006351470947s
Received healthy response to inference request in 2.614125967025757s
Received healthy response to inference request in 2.5746500492095947s
Received healthy response to inference request in 2.273850440979004s
Received healthy response to inference request in 2.9843380451202393s
Received healthy response to inference request in 2.6712143421173096s
Received healthy response to inference request in 2.3907339572906494s
Received healthy response to inference request in 2.340947389602661s
Received healthy response to inference request in 2.4138522148132324s
Received healthy response to inference request in 2.9750499725341797s
Received healthy response to inference request in 2.5216546058654785s
Received healthy response to inference request in 2.579116106033325s
Received healthy response to inference request in 2.4896154403686523s
Received healthy response to inference request in 2.5203731060028076s
Received healthy response to inference request in 2.471825361251831s
Received healthy response to inference request in 2.5121262073516846s
Received healthy response to inference request in 2.5314462184906006s
Received healthy response to inference request in 2.390565872192383s
Received healthy response to inference request in 2.786769151687622s
Received healthy response to inference request in 2.509092092514038s
Received healthy response to inference request in 2.6611790657043457s
Received healthy response to inference request in 2.3450722694396973s
Received healthy response to inference request in 2.670621395111084s
Received healthy response to inference request in 2.5076534748077393s
Received healthy response to inference request in 2.3723666667938232s
Received healthy response to inference request in 2.605958938598633s
Received healthy response to inference request in 2.699819326400757s
Received healthy response to inference request in 2.324873924255371s
Received healthy response to inference request in 2.7140345573425293s
Received healthy response to inference request in 2.486469030380249s
Received healthy response to inference request in 2.4434332847595215s
Received healthy response to inference request in 2.525662660598755s
Inference service chaiml-235b-sft-prod-rm-5716-v1 ready after 1033.2461676597595s
Pipeline stage VLLMDeployer completed in 1038.84s
Received healthy response to inference request in 2.814492702484131s
Received healthy response to inference request in 3.30747652053833s
run pipeline stage %s
Received healthy response to inference request in 2.6900644302368164s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.453214168548584s
Received healthy response to inference request in 2.4743316173553467s
Received healthy response to inference request in 2.719390630722046s
Received healthy response to inference request in 2.439971685409546s
Received healthy response to inference request in 3.434523582458496s
Received healthy response to inference request in 3.005479097366333s
Received healthy response to inference request in 2.4863054752349854s
Received healthy response to inference request in 2.8921689987182617s
Received healthy response to inference request in 2.545168161392212s
Received healthy response to inference request in 2.4312243461608887s
Received healthy response to inference request in 2.490976333618164s
Received healthy response to inference request in 2.3097805976867676s
Received healthy response to inference request in 2.5719151496887207s
Received healthy response to inference request in 2.3669979572296143s
Received healthy response to inference request in 2.3605690002441406s
Received healthy response to inference request in 2.2936267852783203s
Received healthy response to inference request in 2.427171230316162s
Received healthy response to inference request in 2.4105446338653564s
Received healthy response to inference request in 2.667022943496704s
Received healthy response to inference request in 2.526991844177246s
Received healthy response to inference request in 2.610781192779541s
Received healthy response to inference request in 2.429600954055786s
Received healthy response to inference request in 2.704782724380493s
Received healthy response to inference request in 2.4698243141174316s
Received healthy response to inference request in 3.1993415355682373s
Received healthy response to inference request in 2.438025712966919s
Received healthy response to inference request in 2.5215272903442383s
Received healthy response to inference request in 2.254854440689087s
30 requests
0 failed requests
Received healthy response to inference request in 2.56390643119812s
5th percentile: 2.37760351896286
Received healthy response to inference request in 2.3394289016723633s
Received healthy response to inference request in 2.832764148712158s
10th percentile: 2.4085467576980593
20th percentile: 2.430899667739868
Received healthy response to inference request in 2.4845848083496094s
30th percentile: 2.473579740524292
Received healthy response to inference request in 2.573455572128296s
Received healthy response to inference request in 2.31477952003479s
40th percentile: 2.486458349227905
50th percentile: 2.5416829586029053
Received healthy response to inference request in 2.5816333293914795s
60th percentile: 2.661506175994873
30 requests
Received healthy response to inference request in 3.133693218231201s
Received healthy response to inference request in 2.815133810043335s
70th percentile: 2.679795837402344
0 failed requests
80th percentile: 2.7423322200775146
5th percentile: 2.333963179588318
90th percentile: 2.8475175857543946
Received healthy response to inference request in 2.6177892684936523s
Received healthy response to inference request in 2.493074417114258s
10th percentile: 2.3696372270584107
95th percentile: 3.1715776801109303
20th percentile: 2.4235371112823487
99th percentile: 3.362178783416748
30th percentile: 2.4525166511535645
Received healthy response to inference request in 2.5423080921173096s
mean time: 2.6219982226689655
Received healthy response to inference request in 2.4958608150482178s
40th percentile: 2.487603187561035
Pipeline stage StressChecker completed in 141.16s
30 requests
50th percentile: 2.5325329303741455
run pipeline stage %s
0 failed requests
Received healthy response to inference request in 2.5351736545562744s
60th percentile: 2.5758024215698243
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
5th percentile: 2.3383435368537904
70th percentile: 2.6182520389556885
run_pipeline:run_in_cloud %s
10th percentile: 2.3586068391799926
80th percentile: 2.792313861846924
Received healthy response to inference request in 2.3737430572509766s
starting trigger_guanaco_pipeline args=%s
20th percentile: 2.4791653633117674
90th percentile: 2.9950837373733523
30th percentile: 2.49502489566803
triggered trigger_guanaco_pipeline args=%s
95th percentile: 3.188562273979187
40th percentile: 2.521065616607666
Received healthy response to inference request in 2.6555888652801514s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 7.95s
99th percentile: 3.366320788860321
50th percentile: 2.5285544395446777
Shutdown handler de-registered
mean time: 2.620049301783244
60th percentile: 2.6092257499694824
chaiml-235b-sft-prod-rm_18463_v1 status is now deployed due to DeploymentManager action
Pipeline stage StressChecker completed in 142.93s
70th percentile: 2.668102478981018
Received healthy response to inference request in 2.8208582401275635s
run pipeline stage %s
80th percentile: 2.6930080890655517
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
90th percentile: 2.728964948654175
run_pipeline:run_in_cloud %s
95th percentile: 2.8248304963111877
Received healthy response to inference request in 2.247988224029541s
starting trigger_guanaco_pipeline args=%s
99th percentile: 2.940381615161896
mean time: 2.574178210894267
triggered trigger_guanaco_pipeline args=%s
Pipeline stage StressChecker completed in 139.03s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 5.25s
Received healthy response to inference request in 2.667527198791504s
run pipeline stage %s
Shutdown handler de-registered
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
chaiml-235b-sft-prod-rm_88661_v1 status is now deployed due to DeploymentManager action
chaiml-235b-sft-prod-rm_88661_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-235b-sft-prod-rm_88661_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-235b-sft-prod-rm-88661-v1 is running
Tearing down inference service chaiml-235b-sft-prod-rm-88661-v1
Service chaiml-235b-sft-prod-rm-88661-v1 has been torndown
Pipeline stage VLLMDeleter completed in 0.99s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
clean up pipeline due to error=TeardownError("Got unexpected keyword argument 'request_checksum_calculation'")
Shutdown handler de-registered
chaiml-235b-sft-prod-rm_88661_v1 status is now torndown due to DeploymentManager action