chaiml-235b-sft-prod-rm_63215

developer_uid: rirv938
submission_id: chaiml-235b-sft-prod-rm_63215_v1
model_name: chaiml-235b-sft-prod-rm_63215_v1
model_group: ChaiML/235b_sft_prod_rm_
status: torndown
timestamp: 2026-02-22T21:06:49+00:00
num_battles: 10550
num_wins: 5750
celo_rating: 1329.14
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-235b-sft-prod-rm_63215_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-16
win_ratio: 0.5450236966824644
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_start|>', '</s>', 'You:', '<|im_end|>', '###'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-prod-rm-63215-v1-uploader
Waiting for job on chaiml-235b-sft-prod-rm-63215-v1-uploader to finish
chaiml-235b-sft-prod-rm-5716-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-5716-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e5-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-5716-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e5-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-5716-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-5716-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e5-W4A16...
chaiml-235b-sft-prod-rm-88661-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-88661-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-88661-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-88661-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-88661-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16...
chaiml-235b-sft-prod-rm-18463-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e3-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-63215-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-18463-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e3-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-63215-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-18463-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-63215-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-18463-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e3-W4A16...
chaiml-235b-sft-prod-rm-63215-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-63215-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16...
chaiml-235b-sft-prod-rm-38783-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-38783-v1-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-38783-v1-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-38783-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-38783-v1-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16...
chaiml-235b-sft-prod-rm-5716-v1-uploader: Downloaded in 52.690s
chaiml-235b-sft-prod-rm-5716-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e5-W4A16 in 53.337s
chaiml-235b-sft-prod-rm-18463-v1-uploader: Downloaded in 51.253s
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/.gitattributes
chaiml-235b-sft-prod-rm-18463-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e3-W4A16 in 51.899s
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/chat_template.jinja
chaiml-235b-sft-prod-rm-18463-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/generation_config.json
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-18463-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/added_tokens.json
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-18463-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/quantization_config.json
chaiml-235b-sft-prod-rm-38783-v1-uploader: Downloaded in 52.351s
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/config.json
chaiml-235b-sft-prod-rm-38783-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e2-W4A16 in 53.016s
chaiml-235b-sft-prod-rm-18463-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/merges.txt
chaiml-235b-sft-prod-rm-88661-v1-uploader: Downloaded in 53.115s
chaiml-235b-sft-prod-rm-38783-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/tokenizer.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 in 53.672s
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-38783-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/vocab.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-88661-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-38783-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-88661-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-38783-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-88661-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-18463-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-88661-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-63215-v1-uploader: Downloaded in 53.310s
chaiml-235b-sft-prod-rm-18463-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-63215-v1-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 in 53.868s
chaiml-235b-sft-prod-rm-18463-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default
chaiml-235b-sft-prod-rm-88661-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-38783-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-63215-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/.gitattributes
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/added_tokens.json
chaiml-235b-sft-prod-rm-88661-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-38783-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-63215-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/chat_template.jinja
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-38783-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-88661-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-38783-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-63215-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-63215-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-88661-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default
chaiml-235b-sft-prod-rm-63215-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/added_tokens.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/chat_template.jinja
chaiml-235b-sft-prod-rm-63215-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/.gitattributes
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/generation_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/quantization_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/merges.txt
chaiml-235b-sft-prod-rm-63215-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/vocab.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/added_tokens.json
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/tokenizer.json
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/tokenizer.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/generation_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/.gitattributes
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/chat_template.jinja
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/quantization_config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/config.json
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/merges.txt
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00006-of-00027.safetensors?partNumber=7&uploadId=2~AaVb3XrLHbm6wYTmWy_bfZy9EBj6179": write tcp 10.0.50.49:60586->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00021-of-00027.safetensors?partNumber=6&uploadId=2~R7Uc_sIHOwPAsejkX7hi47o35JLmR76": write tcp 10.0.50.49:60560->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/tokenizer.json
chaiml-235b-sft-prod-rm-38783-v1-uploader: ERROR "cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00021-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~R7Uc_sIHOwPAsejkX7hi47o35JLmR76 caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000092f976f82401fcd-0069935327-156987ee57-default, host id:
chaiml-235b-sft-prod-rm-38783-v1-uploader: ERROR "cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00006-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~AaVb3XrLHbm6wYTmWy_bfZy9EBj6179 caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000ae749a2a989b0feb-0069935327-15656b5c61-default, host id:
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00002-of-00027.safetensors?partNumber=1&uploadId=2~KEjbczxMbDbiE4YPF-Z90M1A7d4SCJy": write tcp 10.0.50.49:60550->216.153.53.63:443: write: connection timed out
chaiml-235b-sft-prod-rm-38783-v1-uploader: ERROR "cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00002-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~KEjbczxMbDbiE4YPF-Z90M1A7d4SCJy caused by: SignatureDoesNotMatch:  status code: 403, request id: tx000007728fab36a2414db-00699353c3-15656b5c61-default, host id:
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-38783-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00022-of-00027.safetensors?partNumber=84&uploadId=2~yQRTvmu_rZbCpGDqgUl1YMlm1iOD-h8": write tcp 10.0.50.49:56256->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-prod-rm-38783-v1-uploader: ERROR "cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00022-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~yQRTvmu_rZbCpGDqgUl1YMlm1iOD-h8 caused by: SignatureDoesNotMatch:  status code: 403, request id: tx000004cce0a1fa2be60fb-006993557e-156987ee57-default, host id:
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-5716-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00015-of-00027.safetensors?partNumber=76&uploadId=2~6_pRoXmSyvl3oI9udEQzr94Zu_6hDP3": write tcp 10.0.39.88:50612->216.153.53.63:443: use of closed network connection
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/.gitattributes": object size matches
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-prod-rm-18463-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-18463-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/added_tokens.json": object size matches
chaiml-235b-sft-prod-rm-63215-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-5716-v1-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-235b-sft-prod-rm-88661-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/chat_template.jinja": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/.gitattributes": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/added_tokens.json": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/generation_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/chat_template.jinja": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/merges.txt": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/config.json": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00001-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/generation_config.json": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00003-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/merges.txt": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00004-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00001-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00005-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00002-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00007-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00003-of-00027.safetensors": object size matches
Job chaiml-235b-sft-prod-rm-18463-v1-uploader completed after 937.54s with status: succeeded
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00008-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00004-of-00027.safetensors": object size matches
Job chaiml-235b-sft-prod-rm-63215-v1-uploader completed after 927.59s with status: succeeded
Stopping job with name chaiml-235b-sft-prod-rm-18463-v1-uploader
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00009-of-00027.safetensors": object size matches
Job chaiml-235b-sft-prod-rm-88661-v1-uploader completed after 944.08s with status: succeeded
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00005-of-00027.safetensors": object size matches
Stopping job with name chaiml-235b-sft-prod-rm-63215-v1-uploader
Pipeline stage VLLMUploader completed in 943.07s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00010-of-00027.safetensors": object size matches
Stopping job with name chaiml-235b-sft-prod-rm-88661-v1-uploader
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00006-of-00027.safetensors": object size matches
Pipeline stage VLLMUploader completed in 933.65s
run pipeline stage %s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00011-of-00027.safetensors": object size matches
Pipeline stage VLLMUploader completed in 949.74s
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00007-of-00027.safetensors": object size matches
run pipeline stage %s
Running pipeline stage VLLMTemplater
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00012-of-00027.safetensors": object size matches
run pipeline stage %s
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00008-of-00027.safetensors": object size matches
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.90s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00013-of-00027.safetensors": object size matches
Running pipeline stage VLLMTemplater
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00009-of-00027.safetensors": object size matches
run pipeline stage %s
Pipeline stage VLLMTemplater completed in 1.70s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00014-of-00027.safetensors": object size matches
Pipeline stage VLLMTemplater completed in 2.01s
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00010-of-00027.safetensors": object size matches
Running pipeline stage VLLMDeployer
run pipeline stage %s
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00015-of-00027.safetensors": object size matches
run pipeline stage %s
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00011-of-00027.safetensors": object size matches
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-prod-rm-18463-v1
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00016-of-00027.safetensors": object size matches
Running pipeline stage VLLMDeployer
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00012-of-00027.safetensors": object size matches
Creating inference service chaiml-235b-sft-prod-rm-63215-v1
Waiting for inference service chaiml-235b-sft-prod-rm-18463-v1 to be ready
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00017-of-00027.safetensors": object size matches
Creating inference service chaiml-235b-sft-prod-rm-88661-v1
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00013-of-00027.safetensors": object size matches
Waiting for inference service chaiml-235b-sft-prod-rm-63215-v1 to be ready
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00018-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00014-of-00027.safetensors": object size matches
Waiting for inference service chaiml-235b-sft-prod-rm-88661-v1 to be ready
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00019-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00016-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00020-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00017-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00023-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00018-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00024-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00019-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00025-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00020-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00026-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00021-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model-00027-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00022-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/model.safetensors.index.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00023-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/quantization_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00024-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/special_tokens_map.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00025-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/tokenizer.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00026-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/tokenizer_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model-00027-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-38783-v1-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-38783-v1/default/vocab.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/model.safetensors.index.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/quantization_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/special_tokens_map.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/tokenizer.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/tokenizer_config.json": object size matches
chaiml-235b-sft-prod-rm-5716-v1-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-5716-v1/default/vocab.json": object size matches
Job chaiml-235b-sft-prod-rm-38783-v1-uploader completed after 977.62s with status: succeeded
Stopping job with name chaiml-235b-sft-prod-rm-38783-v1-uploader
Pipeline stage VLLMUploader completed in 981.40s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.50s
Job chaiml-235b-sft-prod-rm-5716-v1-uploader completed after 999.97s with status: succeeded
run pipeline stage %s
Stopping job with name chaiml-235b-sft-prod-rm-5716-v1-uploader
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploader completed in 1004.30s
Creating inference service chaiml-235b-sft-prod-rm-38783-v1
run pipeline stage %s
Waiting for inference service chaiml-235b-sft-prod-rm-38783-v1 to be ready
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.54s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-prod-rm-5716-v1
Waiting for inference service chaiml-235b-sft-prod-rm-5716-v1 to be ready
Inference service chaiml-235b-sft-prod-rm-63215-v1 ready after 780.560982465744s
Pipeline stage VLLMDeployer completed in 786.40s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.9106545448303223s
Received healthy response to inference request in 2.4552035331726074s
Received healthy response to inference request in 2.692019462585449s
Received healthy response to inference request in 2.474421739578247s
Received healthy response to inference request in 2.7645368576049805s
Received healthy response to inference request in 2.5120503902435303s
Received healthy response to inference request in 2.63553524017334s
Received healthy response to inference request in 2.5533299446105957s
Received healthy response to inference request in 2.368685722351074s
Received healthy response to inference request in 2.4482309818267822s
Received healthy response to inference request in 2.7424418926239014s
Received healthy response to inference request in 2.581507921218872s
Received healthy response to inference request in 2.382631778717041s
Received healthy response to inference request in 2.3774311542510986s
Received healthy response to inference request in 2.942600965499878s
Received healthy response to inference request in 2.3730084896087646s
Received healthy response to inference request in 2.4892466068267822s
Received healthy response to inference request in 2.5626676082611084s
Received healthy response to inference request in 2.5661089420318604s
Received healthy response to inference request in 2.7901852130889893s
Received healthy response to inference request in 2.701385498046875s
Received healthy response to inference request in 2.471928834915161s
Received healthy response to inference request in 2.7403275966644287s
Received healthy response to inference request in 2.500352144241333s
Received healthy response to inference request in 2.493736505508423s
Received healthy response to inference request in 2.47042179107666s
Received healthy response to inference request in 2.5407485961914062s
Received healthy response to inference request in 2.4436745643615723s
Received healthy response to inference request in 2.6298956871032715s
Received healthy response to inference request in 2.8829574584960938s
30 requests
0 failed requests
5th percentile: 2.374998688697815
10th percentile: 2.3821117162704466
20th percentile: 2.4538090229034424
30th percentile: 2.473673868179321
40th percentile: 2.497705888748169
50th percentile: 2.547039270401001
60th percentile: 2.572268533706665
70th percentile: 2.6524805068969726
80th percentile: 2.740750455856323
90th percentile: 2.7994624376296997
95th percentile: 2.9157613873481747
99th percentile: 3.6299190068244944
mean time: 2.6165975888570148
Pipeline stage StressChecker completed in 137.36s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 6.10s
Shutdown handler de-registered
chaiml-235b-sft-prod-rm_63215_v1 status is now deployed due to DeploymentManager action
chaiml-235b-sft-prod-rm_63215_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-235b-sft-prod-rm_63215_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-235b-sft-prod-rm-63215-v1 is running
Tearing down inference service chaiml-235b-sft-prod-rm-63215-v1
Service chaiml-235b-sft-prod-rm-63215-v1 has been torndown
Pipeline stage VLLMDeleter completed in 1.07s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
clean up pipeline due to error=TeardownError("Got unexpected keyword argument 'request_checksum_calculation'")
Shutdown handler de-registered
chaiml-235b-sft-prod-rm_63215_v1 status is now torndown due to DeploymentManager action