chaiml-235b-sft-new-rm-_42002

developer_uid: rirv938
submission_id: chaiml-235b-sft-new-rm-_42002_v2
model_name: chaiml-235b-sft-new-rm-_42002_v2
model_group: ChaiML/235b_sft_new_rm_l
status: torndown
timestamp: 2026-02-22T21:42:01+00:00
num_battles: 10384
num_wins: 5181
celo_rating: 1296.48
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 1
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-235b-sft-new-rm-_42002_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-16
win_ratio: 0.4989406779661017
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['###', '</s>', '<|im_end|>', 'You:', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 1, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-42002-v2-uploader
Waiting for job on chaiml-235b-sft-new-rm-42002-v2-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-68639-v2-uploader
Waiting for job on chaiml-235b-sft-new-rm-68639-v2-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-prod-rm-88661-v2-uploader
Waiting for job on chaiml-235b-sft-prod-rm-88661-v2-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-prod-rm-63215-v2-uploader
Waiting for job on chaiml-235b-sft-prod-rm-63215-v2-uploader to finish
chaiml-235b-sft-prod-rm-88661-v2-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-88661-v2-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-88661-v2-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-88661-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-88661-v2-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16...
chaiml-235b-sft-new-rm-42002-v2-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-42002-v2-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-42002-v2-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-42002-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-42002-v2-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16...
chaiml-235b-sft-prod-rm-63215-v2-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-68639-v2-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-prod-rm-63215-v2-uploader: Repo ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-68639-v2-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-prod-rm-63215-v2-uploader: Checking if ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-68639-v2-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 already exists in ChaiML
chaiml-235b-sft-prod-rm-63215-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-68639-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-prod-rm-63215-v2-uploader: Downloading snapshot of ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16...
chaiml-235b-sft-new-rm-68639-v2-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16...
chaiml-235b-sft-prod-rm-88661-v2-uploader: Downloaded in 53.589s
chaiml-235b-sft-prod-rm-88661-v2-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e4-W4A16 in 54.121s
chaiml-235b-sft-new-rm-42002-v2-uploader: Downloaded in 52.044s
chaiml-235b-sft-new-rm-42002-v2-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16 in 52.697s
chaiml-235b-sft-new-rm-42002-v2-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-42002-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-42002-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-42002-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-42002-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-42002-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-42002-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-42002-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-42002-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-42002-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-42002-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/added_tokens.json
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/chat_template.jinja
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/.gitattributes
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/generation_config.json
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/tokenizer_config.json
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/special_tokens_map.json
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/quantization_config.json
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/config.json
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/vocab.json
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/merges.txt
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/tokenizer.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: Downloaded in 52.512s
chaiml-235b-sft-new-rm-68639-v2-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-prod-rm-63215-v2-uploader: Processed model ChaiML/235b_sft_prod_rm_lexical_10k_kimi_3k_e1-W4A16 in 53.110s
chaiml-235b-sft-new-rm-68639-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-63215-v2-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-68639-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-prod-rm-63215-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-prod-rm-63215-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-68639-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-prod-rm-63215-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-68639-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-63215-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-68639-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-63215-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-63215-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-68639-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-prod-rm-63215-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-63215-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-68639-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-63215-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-prod-rm-63215-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-68639-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-prod-rm-63215-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-prod-rm-63215-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-68639-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-prod-rm-63215-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-68639-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-prod-rm-63215-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-68639-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-prod-rm-63215-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-68639-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-prod-rm-63215-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-68639-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default
chaiml-235b-sft-prod-rm-63215-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/.gitattributes
chaiml-235b-sft-prod-rm-63215-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/added_tokens.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/added_tokens.json
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/chat_template.jinja
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/generation_config.json
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/tokenizer_config.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/.gitattributes
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/special_tokens_map.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/chat_template.jinja
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/generation_config.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/special_tokens_map.json
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/config.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/tokenizer_config.json
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/quantization_config.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/config.json
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/vocab.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/quantization_config.json
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/tokenizer.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/vocab.json
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model.safetensors.index.json
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/merges.txt
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/tokenizer.json
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00027-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-prod-rm-63215-v2-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00013-of-00027.safetensors?partNumber=50&uploadId=2~bgz9kxvbWhoY7NM1RJkKff9OQ7DJlFL": write tcp 10.0.50.56:48318->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-prod-rm-63215-v2-uploader: ERROR "cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00013-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~bgz9kxvbWhoY7NM1RJkKff9OQ7DJlFL caused by: SignatureDoesNotMatch:  status code: 403, request id: tx000009b5e159f8e454983-0069935d3d-15656b5c61-default, host id:
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00006-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00002-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00014-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00020-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00002-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00001-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00021-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00003-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00003-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00009-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00002-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00003-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00025-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00021-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00024-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00006-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00010-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00005-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00003-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00020-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00011-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00022-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00026-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00013-of-00027.safetensors
chaiml-235b-sft-prod-rm-88661-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-88661-v2/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v2/default/model-00007-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00023-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00001-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00015-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00014-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00002-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00024-of-00027.safetensors
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v2/default/model-00022-of-00027.safetensors
Job chaiml-235b-sft-prod-rm-88661-v2-uploader completed after 747.8s with status: succeeded
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00010-of-00027.safetensors
Stopping job with name chaiml-235b-sft-prod-rm-88661-v2-uploader
chaiml-235b-sft-prod-rm-63215-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00005-of-00027.safetensors
Pipeline stage VLLMUploader completed in 751.73s
chaiml-235b-sft-prod-rm-63215-v2-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
run pipeline stage %s
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/.gitattributes": object size matches
Running pipeline stage VLLMTemplater
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/added_tokens.json": object size matches
Pipeline stage VLLMTemplater completed in 0.84s
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/chat_template.jinja": object size matches
run pipeline stage %s
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/config.json": object size matches
Running pipeline stage VLLMDeployer
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/generation_config.json": object size matches
Job chaiml-235b-sft-new-rm-42002-v2-uploader completed after 764.78s with status: succeeded
Creating inference service chaiml-235b-sft-prod-rm-88661-v2
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/merges.txt": object size matches
Stopping job with name chaiml-235b-sft-new-rm-42002-v2-uploader
Waiting for inference service chaiml-235b-sft-prod-rm-88661-v2 to be ready
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00001-of-00027.safetensors": object size matches
Pipeline stage VLLMUploader completed in 767.33s
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00002-of-00027.safetensors": object size matches
run pipeline stage %s
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00003-of-00027.safetensors": object size matches
Running pipeline stage VLLMTemplater
Job chaiml-235b-sft-new-rm-68639-v2-uploader completed after 763.72s with status: succeeded
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00004-of-00027.safetensors": object size matches
Pipeline stage VLLMTemplater completed in 0.87s
Stopping job with name chaiml-235b-sft-new-rm-68639-v2-uploader
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00005-of-00027.safetensors": object size matches
run pipeline stage %s
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00006-of-00027.safetensors": object size matches
Pipeline stage VLLMUploader completed in 766.44s
Running pipeline stage VLLMDeployer
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00007-of-00027.safetensors": object size matches
run pipeline stage %s
Creating inference service chaiml-235b-sft-new-rm-42002-v2
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00008-of-00027.safetensors": object size matches
Running pipeline stage VLLMTemplater
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00009-of-00027.safetensors": object size matches
Waiting for inference service chaiml-235b-sft-new-rm-42002-v2 to be ready
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00010-of-00027.safetensors": object size matches
Pipeline stage VLLMTemplater completed in 1.55s
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00011-of-00027.safetensors": object size matches
run pipeline stage %s
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00012-of-00027.safetensors": object size matches
Running pipeline stage VLLMDeployer
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00014-of-00027.safetensors": object size matches
Creating inference service chaiml-235b-sft-new-rm-68639-v2
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00015-of-00027.safetensors": object size matches
Waiting for inference service chaiml-235b-sft-new-rm-68639-v2 to be ready
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00016-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00017-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00018-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00019-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00020-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00021-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00022-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00023-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00024-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00025-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00026-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model-00027-of-00027.safetensors": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/model.safetensors.index.json": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/quantization_config.json": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/special_tokens_map.json": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/tokenizer.json": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/tokenizer_config.json": object size matches
chaiml-235b-sft-prod-rm-63215-v2-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-prod-rm-63215-v2/default/vocab.json": object size matches
Job chaiml-235b-sft-prod-rm-63215-v2-uploader completed after 787.53s with status: succeeded
Stopping job with name chaiml-235b-sft-prod-rm-63215-v2-uploader
Pipeline stage VLLMUploader completed in 790.69s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.96s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-prod-rm-63215-v2
Waiting for inference service chaiml-235b-sft-prod-rm-63215-v2 to be ready
Inference service chaiml-235b-sft-prod-rm-88661-v2 ready after 648.7243916988373s
Pipeline stage VLLMDeployer completed in 652.49s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.958946943283081s
Received healthy response to inference request in 1.2617440223693848s
Received healthy response to inference request in 1.5354177951812744s
Received healthy response to inference request in 1.4760589599609375s
Received healthy response to inference request in 1.5412561893463135s
Received healthy response to inference request in 1.3252642154693604s
Received healthy response to inference request in 1.4752495288848877s
Received healthy response to inference request in 1.5527734756469727s
Received healthy response to inference request in 1.5361137390136719s
Received healthy response to inference request in 1.4175920486450195s
Received healthy response to inference request in 1.533085823059082s
Received healthy response to inference request in 1.7077765464782715s
Received healthy response to inference request in 1.474433422088623s
Received healthy response to inference request in 1.6012632846832275s
Received healthy response to inference request in 1.4766335487365723s
Received healthy response to inference request in 1.99894380569458s
Received healthy response to inference request in 1.5420489311218262s
Received healthy response to inference request in 1.536926507949829s
Received healthy response to inference request in 1.3927526473999023s
Received healthy response to inference request in 1.5458707809448242s
Received healthy response to inference request in 1.4867184162139893s
Received healthy response to inference request in 1.559112548828125s
Received healthy response to inference request in 1.6642284393310547s
Received healthy response to inference request in 1.6548082828521729s
Received healthy response to inference request in 1.5766618251800537s
Received healthy response to inference request in 1.4225962162017822s
Received healthy response to inference request in 1.5463428497314453s
Received healthy response to inference request in 1.540863037109375s
Received healthy response to inference request in 2.097440004348755s
Received healthy response to inference request in 1.5502231121063232s
30 requests
0 failed requests
5th percentile: 1.3556340098381043
10th percentile: 1.4151081085205077
20th percentile: 1.4750863075256349
30th percentile: 1.4836929559707641
40th percentile: 1.535835361480713
50th percentile: 1.5410596132278442
60th percentile: 1.5460596084594727
70th percentile: 1.5546751976013184
80th percentile: 1.6119722843170168
90th percentile: 1.7328935861587529
95th percentile: 1.9809452176094053
99th percentile: 2.068876106739044
mean time: 1.566304898262024
Pipeline stage StressChecker completed in 89.27s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 4.09s
Shutdown handler de-registered
chaiml-235b-sft-prod-rm_88661_v2 status is now deployed due to DeploymentManager action
Inference service chaiml-235b-sft-new-rm-42002-v2 ready after 780.4621911048889s
Pipeline stage VLLMDeployer completed in 784.43s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.683643102645874s
Received healthy response to inference request in 1.592254877090454s
Received healthy response to inference request in 1.4852359294891357s
Received healthy response to inference request in 2.0702342987060547s
Received healthy response to inference request in 1.5383331775665283s
Received healthy response to inference request in 1.552781105041504s
Received healthy response to inference request in 1.474930763244629s
Received healthy response to inference request in 1.4589717388153076s
Received healthy response to inference request in 1.6355540752410889s
Received healthy response to inference request in 1.4897871017456055s
Received healthy response to inference request in 1.5591397285461426s
Received healthy response to inference request in 1.5475568771362305s
Received healthy response to inference request in 1.5475080013275146s
Received healthy response to inference request in 1.5069670677185059s
Received healthy response to inference request in 1.557447910308838s
Received healthy response to inference request in 1.4779503345489502s
Received healthy response to inference request in 1.549597978591919s
Received healthy response to inference request in 1.5505046844482422s
Inference service chaiml-235b-sft-new-rm-68639-v2 ready after 820.793892621994s
Received healthy response to inference request in 1.5903632640838623s
Pipeline stage VLLMDeployer completed in 824.84s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.4947075843811035s
Received healthy response to inference request in 1.580334186553955s
Received healthy response to inference request in 1.5623526573181152s
Received healthy response to inference request in 1.3833353519439697s
Received healthy response to inference request in 1.576406717300415s
Received healthy response to inference request in 1.5778532028198242s
Received healthy response to inference request in 1.489330768585205s
Received healthy response to inference request in 1.587822675704956s
Received healthy response to inference request in 1.6139090061187744s
Received healthy response to inference request in 1.6741852760314941s
Received healthy response to inference request in 1.5577220916748047s
Received healthy response to inference request in 1.4993407726287842s
Received healthy response to inference request in 1.5249526500701904s
Received healthy response to inference request in 1.5917506217956543s
Received healthy response to inference request in 1.5506341457366943s
Received healthy response to inference request in 1.5639758110046387s
Received healthy response to inference request in 1.475407600402832s
Received healthy response to inference request in 1.4062387943267822s
Received healthy response to inference request in 1.5499296188354492s
Received healthy response to inference request in 1.574068307876587s
Received healthy response to inference request in 1.5241096019744873s
30 requests
0 failed requests
Received healthy response to inference request in 1.5142385959625244s
5th percentile: 1.4751453399658203
10th percentile: 1.4776960611343384
20th percentile: 1.4896958351135254
30th percentile: 1.518966841697693
Received healthy response to inference request in 1.5574054718017578s
40th percentile: 1.5438380718231202
50th percentile: 1.549763798713684
60th percentile: 1.5514929294586182
Received healthy response to inference request in 1.8197884559631348s
70th percentile: 1.558147382736206
80th percentile: 1.5791980266571046
90th percentile: 1.616073513031006
Received healthy response to inference request in 1.522995948791504s
95th percentile: 1.6620030403137205
99th percentile: 1.9581228518486027
mean time: 1.559607481956482
Received healthy response to inference request in 1.6295700073242188s
Pipeline stage StressChecker completed in 81.42s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Received healthy response to inference request in 1.4722535610198975s
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.0467166900634766s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 4.08s
Shutdown handler de-registered
chaiml-235b-sft-new-rm-_42002_v2 status is now deployed due to DeploymentManager action
chaiml-235b-sft-new-rm-_42002_v2 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-235b-sft-new-rm-_42002_v2
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-235b-sft-new-rm-42002-v2 is running
Tearing down inference service chaiml-235b-sft-new-rm-42002-v2
Service chaiml-235b-sft-new-rm-42002-v2 has been torndown
Pipeline stage VLLMDeleter completed in 0.81s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
clean up pipeline due to error=TeardownError("Got unexpected keyword argument 'request_checksum_calculation'")
Shutdown handler de-registered
chaiml-235b-sft-new-rm-_42002_v2 status is now torndown due to DeploymentManager action