chaiml-235b-sft-new-rm-_42002

developer_uid: rirv938
submission_id: chaiml-235b-sft-new-rm-_42002_v1
model_name: chaiml-235b-sft-new-rm-_42002_v1
model_group: ChaiML/235b_sft_new_rm_l
status: torndown
timestamp: 2026-02-22T18:12:06+00:00
num_battles: 10966
num_wins: 6055
celo_rating: 1334.53
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-235b-sft-new-rm-_42002_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-15
win_ratio: 0.5521612256064199
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '</s>', '<|im_start|>', '###', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-42002-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-42002-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-21184-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-21184-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-19484-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-19484-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-68639-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-68639-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-74662-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-74662-v1-uploader to finish
chaiml-235b-sft-new-rm-42002-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-19484-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-19484-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-19484-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-19484-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-19484-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16...
chaiml-235b-sft-new-rm-21184-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-21184-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-21184-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-21184-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-21184-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16...
chaiml-235b-sft-new-rm-74662-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-74662-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-74662-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-74662-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-74662-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16...
chaiml-235b-sft-new-rm-68639-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-68639-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-68639-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-68639-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-68639-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16...
chaiml-235b-sft-new-rm-42002-v1-uploader: Downloaded in 42.416s
chaiml-235b-sft-new-rm-42002-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16 in 42.949s
chaiml-235b-sft-new-rm-42002-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: Downloaded in 40.938s
chaiml-235b-sft-new-rm-42002-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-19484-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 in 41.624s
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-19484-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-42002-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v1-uploader: Downloaded in 41.633s
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-19484-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-68639-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 in 42.164s
chaiml-235b-sft-new-rm-42002-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: Downloaded in 41.836s
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-21184-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 in 42.376s
chaiml-235b-sft-new-rm-19484-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-42002-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-21184-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-42002-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default
chaiml-235b-sft-new-rm-21184-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-19484-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-21184-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-19484-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-74662-v1-uploader: Downloaded in 42.491s
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/generation_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default
chaiml-235b-sft-new-rm-74662-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 in 43.074s
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/.gitattributes
chaiml-235b-sft-new-rm-21184-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-74662-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/config.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/generation_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/merges.txt
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/vocab.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-21184-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-21184-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/vocab.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-21184-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default
chaiml-235b-sft-new-rm-74662-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/generation_config.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-74662-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/vocab.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/generation_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-new-rm-19484-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors?partNumber=74&uploadId=2~KnzMM_iHrH6pMtMPh-RsoJxVK81EBY-": write tcp 10.0.27.120:42634->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: ERROR "cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~KnzMM_iHrH6pMtMPh-RsoJxVK81EBY- caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000858b68de70f88057-006992a26a-1579ff623c-default, host id:
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00001-of-00027.safetensors
Job chaiml-235b-sft-new-rm-42002-v1-uploader completed after 977.9s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-42002-v1-uploader
Pipeline stage VLLMUploader completed in 979.90s
run pipeline stage %s
Running pipeline stage VLLMTemplater
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00020-of-00027.safetensors
Pipeline stage VLLMTemplater completed in 0.70s
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00024-of-00027.safetensors
run pipeline stage %s
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00007-of-00027.safetensors
Running pipeline stage VLLMDeployer
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00002-of-00027.safetensors
Creating inference service chaiml-235b-sft-new-rm-42002-v1
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00004-of-00027.safetensors
Waiting for inference service chaiml-235b-sft-new-rm-42002-v1 to be ready
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00017-of-00027.safetensors
Job chaiml-235b-sft-new-rm-68639-v1-uploader completed after 992.0s with status: succeeded
Job chaiml-235b-sft-new-rm-21184-v1-uploader completed after 1001.88s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-68639-v1-uploader
Stopping job with name chaiml-235b-sft-new-rm-21184-v1-uploader
Pipeline stage VLLMUploader completed in 994.71s
Pipeline stage VLLMUploader completed in 1004.07s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.74s
Job chaiml-235b-sft-new-rm-74662-v1-uploader completed after 990.29s with status: succeeded
Pipeline stage VLLMTemplater completed in 0.77s
run pipeline stage %s
Stopping job with name chaiml-235b-sft-new-rm-74662-v1-uploader
run pipeline stage %s
Running pipeline stage VLLMDeployer
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploader completed in 993.75s
Creating inference service chaiml-235b-sft-new-rm-68639-v1
Creating inference service chaiml-235b-sft-new-rm-21184-v1
run pipeline stage %s
Waiting for inference service chaiml-235b-sft-new-rm-68639-v1 to be ready
Running pipeline stage VLLMTemplater
Waiting for inference service chaiml-235b-sft-new-rm-21184-v1 to be ready
Pipeline stage VLLMTemplater completed in 0.89s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-new-rm-74662-v1
Waiting for inference service chaiml-235b-sft-new-rm-74662-v1 to be ready
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors
Job chaiml-235b-sft-new-rm-19484-v1-uploader completed after 1036.9s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-19484-v1-uploader
Pipeline stage VLLMUploader completed in 1039.33s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.75s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-new-rm-19484-v1
Waiting for inference service chaiml-235b-sft-new-rm-19484-v1 to be ready
Inference service chaiml-235b-sft-new-rm-21184-v1 ready after 284.21195912361145s
Pipeline stage VLLMDeployer completed in 287.49s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.110910654067993s
Received healthy response to inference request in 2.182664394378662s
Received healthy response to inference request in 2.320479154586792s
Received healthy response to inference request in 2.2272443771362305s
Received healthy response to inference request in 2.196024179458618s
Received healthy response to inference request in 2.432786464691162s
Received healthy response to inference request in 2.1420376300811768s
Received healthy response to inference request in 2.075230598449707s
Received healthy response to inference request in 2.3608663082122803s
Received healthy response to inference request in 2.5839426517486572s
Received healthy response to inference request in 1.9517178535461426s
Received healthy response to inference request in 2.15575909614563s
Received healthy response to inference request in 2.4736087322235107s
Received healthy response to inference request in 2.333919048309326s
Received healthy response to inference request in 2.1538188457489014s
Received healthy response to inference request in 2.025940179824829s
Received healthy response to inference request in 2.0790276527404785s
Received healthy response to inference request in 2.042165756225586s
Received healthy response to inference request in 2.234696865081787s
Received healthy response to inference request in 2.1628029346466064s
Received healthy response to inference request in 2.5693507194519043s
Received healthy response to inference request in 2.2414157390594482s
Received healthy response to inference request in 1.9806458950042725s
Received healthy response to inference request in 2.428105592727661s
Received healthy response to inference request in 2.4217584133148193s
Received healthy response to inference request in 1.9162476062774658s
Received healthy response to inference request in 2.3511404991149902s
Received healthy response to inference request in 2.0151288509368896s
Received healthy response to inference request in 2.1614203453063965s
Received healthy response to inference request in 2.088926076889038s
30 requests
0 failed requests
5th percentile: 1.964735472202301
10th percentile: 2.011680555343628
20th percentile: 2.0686176300048826
30th percentile: 2.126104164123535
40th percentile: 2.15915584564209
50th percentile: 2.18934428691864
60th percentile: 2.2373844146728517
70th percentile: 2.3390854835510253
80th percentile: 2.423027849197388
90th percentile: 2.4831829309463505
95th percentile: 2.5773762822151185
99th percentile: 2.958089933395386
mean time: 2.247326103846232
Pipeline stage StressChecker completed in 101.54s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 4.31s
Shutdown handler de-registered
chaiml-235b-sft-new-rm-_21184_v1 status is now deployed due to DeploymentManager action
Inference service chaiml-235b-sft-new-rm-42002-v1 ready after 994.670592546463s
Pipeline stage VLLMDeployer completed in 998.02s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.0977299213409424s
Received healthy response to inference request in 2.399458408355713s
Received healthy response to inference request in 2.5763421058654785s
Received healthy response to inference request in 2.858030319213867s
Received healthy response to inference request in 2.44118595123291s
Received healthy response to inference request in 2.8757450580596924s
Received healthy response to inference request in 2.8443284034729004s
Received healthy response to inference request in 2.5446155071258545s
Received healthy response to inference request in 2.5604970455169678s
Received healthy response to inference request in 2.429673910140991s
Received healthy response to inference request in 2.3868467807769775s
Received healthy response to inference request in 2.496626853942871s
Received healthy response to inference request in 2.4134621620178223s
Received healthy response to inference request in 2.3911209106445312s
Received healthy response to inference request in 2.56704044342041s
Received healthy response to inference request in 2.476635217666626s
Received healthy response to inference request in 2.3877668380737305s
Received healthy response to inference request in 2.4317421913146973s
Received healthy response to inference request in 2.619509696960449s
Received healthy response to inference request in 2.391740083694458s
Received healthy response to inference request in 2.451599597930908s
Received healthy response to inference request in 2.3814611434936523s
Received healthy response to inference request in 2.496241807937622s
Received healthy response to inference request in 2.6144020557403564s
Received healthy response to inference request in 2.5028927326202393s
Received healthy response to inference request in 2.4484102725982666s
Received healthy response to inference request in 2.4050216674804688s
Received healthy response to inference request in 2.511706590652466s
Received healthy response to inference request in 2.409762144088745s
Received healthy response to inference request in 2.383235454559326s
30 requests
0 failed requests
5th percentile: 2.3848605513572694
10th percentile: 2.387674832344055
20th percentile: 2.397914743423462
30th percentile: 2.4123521566390993
40th percentile: 2.437408447265625
50th percentile: 2.464117407798767
60th percentile: 2.4991332054138184
70th percentile: 2.5493799686431884
80th percentile: 2.583954095840454
90th percentile: 2.845698595046997
95th percentile: 2.867773425579071
99th percentile: 3.03335431098938
mean time: 2.5264943758646647
Pipeline stage StressChecker completed in 112.65s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.81s
Shutdown handler de-registered
chaiml-235b-sft-new-rm-_42002_v1 status is now deployed due to DeploymentManager action
chaiml-235b-sft-new-rm-_42002_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-235b-sft-new-rm-_42002_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-235b-sft-new-rm-42002-v1 is running
Tearing down inference service chaiml-235b-sft-new-rm-42002-v1
Service chaiml-235b-sft-new-rm-42002-v1 has been torndown
Pipeline stage VLLMDeleter completed in 0.90s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
clean up pipeline due to error=TeardownError("Got unexpected keyword argument 'request_checksum_calculation'")
Shutdown handler de-registered
chaiml-235b-sft-new-rm-_42002_v1 status is now torndown due to DeploymentManager action